[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:18 dist-test-slave-w4v5 krb5kdc[3356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:18 dist-test-slave-w4v5 krb5kdc[3356](info): set up 2 sockets
Aug 06 09:10:18 dist-test-slave-w4v5 krb5kdc[3356](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 09:10:20.050454 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:20 dist-test-slave-w4v5 krb5kdc[3356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:20.248066 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:44121
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:44121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:20.632843 3373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:20.633178 3373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:20.633335 3373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:20.640776 3373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:20.640880 3373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:20.640928 3373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:20.640988 3373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:20.641027 3373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:20.651101 3373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:44121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:20.652789 3373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:20.654580 3373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:20.679909 3380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:20.680035 3383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:20.680007 3381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:20.682428 3373 server_base.cc:1034] running on GCE node
I20260806 09:10:20.683669 3373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:20.685899 3373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:20.687170 3373 hybrid_clock.cc:648] HybridClock initialized: now 1786007420687046 us; error 107 us; skew 500 ppm
I20260806 09:10:20.687721 3373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:20 dist-test-slave-w4v5 krb5kdc[3356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:20.695482 3373 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:20.698135 3373 webserver.cc:460] Webserver started at http://127.3.69.62:39983/ using document root <none> and password file <none>
I20260806 09:10:20.699386 3373 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:20.699553 3373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:20.700065 3373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:20.704571 3373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "afa8b74699874cbd96fe69393d131479"
format_stamp: "Formatted at 2026-08-06 09:10:20 on dist-test-slave-w4v5"
server_key: "99005c202cbafc9dfdc3c7fa92f45439"
server_key_iv: "95a1ae410478b2a97636a5f622d38551"
server_key_version: "encryptionkey@0"
I20260806 09:10:20.705725 3373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "afa8b74699874cbd96fe69393d131479"
format_stamp: "Formatted at 2026-08-06 09:10:20 on dist-test-slave-w4v5"
server_key: "99005c202cbafc9dfdc3c7fa92f45439"
server_key_iv: "95a1ae410478b2a97636a5f622d38551"
server_key_version: "encryptionkey@0"
I20260806 09:10:20.713687 3373 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260806 09:10:20.719045 3392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:20.721103 3373 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260806 09:10:20.721398 3373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "afa8b74699874cbd96fe69393d131479"
format_stamp: "Formatted at 2026-08-06 09:10:20 on dist-test-slave-w4v5"
server_key: "99005c202cbafc9dfdc3c7fa92f45439"
server_key_iv: "95a1ae410478b2a97636a5f622d38551"
server_key_version: "encryptionkey@0"
I20260806 09:10:20.721637 3373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:20.743770 3373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:20.749099 3373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:20.749456 3373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:20.780718 3373 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:44121
I20260806 09:10:20.780803 3462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:44121 every 8 connection(s)
I20260806 09:10:20.783026 3373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:20.789343 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3373
I20260806 09:10:20.789844 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:20.790849 3463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:20.793514 3348 external_mini_cluster.cc:1428] Setting key b32a760a0690d6b7d7e9edd0b8de7e13
I20260806 09:10:20.810456 3463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479: Bootstrap starting.
I20260806 09:10:20.816452 3463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:20.818796 3463 log.cc:824] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:20.823354 3463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479: No bootstrap required, opened a new log
Aug 06 09:10:20 dist-test-slave-w4v5 krb5kdc[3356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:20.835764 3463 raft_consensus.cc:348] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa8b74699874cbd96fe69393d131479" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44121 } }
I20260806 09:10:20.836275 3463 raft_consensus.cc:374] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:20.836485 3463 raft_consensus.cc:729] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa8b74699874cbd96fe69393d131479, State: Initialized, Role: FOLLOWER
I20260806 09:10:20.837632 3463 consensus_queue.cc:260] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa8b74699874cbd96fe69393d131479" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44121 } }
I20260806 09:10:20.837976 3463 raft_consensus.cc:388] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:20.838272 3463 raft_consensus.cc:482] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:20.838575 3463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:20.841142 3463 raft_consensus.cc:504] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa8b74699874cbd96fe69393d131479" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44121 } }
I20260806 09:10:20.842129 3463 leader_election.cc:302] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afa8b74699874cbd96fe69393d131479; no voters:
I20260806 09:10:20.843042 3463 leader_election.cc:288] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:20.843348 3470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:20.845557 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:20.800977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.3.69.62:44121)
0806 09:10:20.802464 (+ 1487us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:20.802484 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:20.805834 (+ 3350us) server_negotiation.cc:407] Connection header received
0806 09:10:20.815778 (+ 9944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:20.815800 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:20.816364 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:20.816795 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:20.821830 (+ 5035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:20.823548 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:20.827010 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:20.827890 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:20.828133 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:20.835371 (+ 7238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:20.835442 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:20.835478 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:20.835518 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:20.839256 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:20.840107 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:20.840119 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:20.840135 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:20.840332 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:20.840892 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:20.840897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:20.840900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:20.841640 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:20.842083 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:20.843142 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0806 09:10:20.843699 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004,"thread_start_us":577,"threads_started":1}
I20260806 09:10:20.847131 3470 raft_consensus.cc:686] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [term 1 LEADER]: Becoming Leader. State: Replica: afa8b74699874cbd96fe69393d131479, State: Running, Role: LEADER
I20260806 09:10:20.848161 3470 consensus_queue.cc:237] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa8b74699874cbd96fe69393d131479" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44121 } }
I20260806 09:10:20.849203 3463 sys_catalog.cc:564] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:20.856621 3473 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afa8b74699874cbd96fe69393d131479. Latest consensus state: current_term: 1 leader_uuid: "afa8b74699874cbd96fe69393d131479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa8b74699874cbd96fe69393d131479" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44121 } } }
I20260806 09:10:20.856714 3472 sys_catalog.cc:455] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afa8b74699874cbd96fe69393d131479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afa8b74699874cbd96fe69393d131479" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44121 } } }
I20260806 09:10:20.857090 3473 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:20.857264 3472 sys_catalog.cc:458] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:20.858453 3488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:20.868405 3488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:20.884268 3488 catalog_manager.cc:1347] Generated new cluster ID: fc9c018c4e524cfaaa065118f9b5019d
I20260806 09:10:20.884402 3488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:20.903662 3488 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:20.905268 3488 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:20.912149 3488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afa8b74699874cbd96fe69393d131479: Generated new TSK 0
I20260806 09:10:20.913367 3488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:10:20.978961 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:44121
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:21.347474 3507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:21.347959 3507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:21.348193 3507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:21.356078 3507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:21.356230 3507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:21.356415 3507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:10:21.366711 3507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:44121
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:21.368058 3507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:21.369483 3507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:21.384137 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:21.384213 3522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:21.384151 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:21.385205 3507 server_base.cc:1034] running on GCE node
I20260806 09:10:21.385912 3507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:21.386967 3507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:21.388240 3507 hybrid_clock.cc:648] HybridClock initialized: now 1786007421388169 us; error 60 us; skew 500 ppm
I20260806 09:10:21.388607 3507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:21 dist-test-slave-w4v5 krb5kdc[3356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:21.392320 3507 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:10:21.393915 3507 webserver.cc:460] Webserver started at http://127.3.69.1:38983/ using document root <none> and password file <none>
I20260806 09:10:21.394652 3507 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:21.394805 3507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:21.395210 3507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:21.398341 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "11abbd3f664e4027882adff6b34c6a86"
format_stamp: "Formatted at 2026-08-06 09:10:21 on dist-test-slave-w4v5"
server_key: "86c4eaefc546914696c2cec9a3b77803"
server_key_iv: "2355dc0cec5e1532f697815a639b6a21"
server_key_version: "encryptionkey@0"
I20260806 09:10:21.399241 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "11abbd3f664e4027882adff6b34c6a86"
format_stamp: "Formatted at 2026-08-06 09:10:21 on dist-test-slave-w4v5"
server_key: "86c4eaefc546914696c2cec9a3b77803"
server_key_iv: "2355dc0cec5e1532f697815a639b6a21"
server_key_version: "encryptionkey@0"
I20260806 09:10:21.404992 3507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:10:21.408744 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:21.410022 3507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:10:21.410205 3507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "11abbd3f664e4027882adff6b34c6a86"
format_stamp: "Formatted at 2026-08-06 09:10:21 on dist-test-slave-w4v5"
server_key: "86c4eaefc546914696c2cec9a3b77803"
server_key_iv: "2355dc0cec5e1532f697815a639b6a21"
server_key_version: "encryptionkey@0"
I20260806 09:10:21.410408 3507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:21.444418 3507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:21.448653 3507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:21.448961 3507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:21.450313 3507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:21.452119 3507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:21.452230 3507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:21.452456 3507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:21.452567 3507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:21.494387 3507 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:46857
I20260806 09:10:21.494424 3701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:46857 every 8 connection(s)
I20260806 09:10:21.495846 3507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:10:21.501271 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3507
I20260806 09:10:21.501569 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:10:21.502213 3348 external_mini_cluster.cc:1428] Setting key aceec0c5ef6cbb6cbce8e4e3899d5229
Aug 06 09:10:21 dist-test-slave-w4v5 krb5kdc[3356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:21.519984 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:21.499168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:44097 (local address 127.3.69.62:44121)
0806 09:10:21.499710 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:21.499717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:21.500805 (+ 1088us) server_negotiation.cc:407] Connection header received
0806 09:10:21.503190 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:21.503196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:21.503282 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:21.503411 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:21.505464 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:21.506869 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:21.508714 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:21.509465 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:21.509642 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:21.513538 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:21.513544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:21.513547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:21.513580 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:21.516528 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:21.517300 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:21.517307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:21.517311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:21.517470 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:21.517925 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:21.517930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:21.517933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:21.518240 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:21.518412 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:21.519501 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0806 09:10:21.519712 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":228,"threads_started":1}
I20260806 09:10:21.521744 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:21.499773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:44121 (local address 127.3.69.1:44097)
0806 09:10:21.500621 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:21.500673 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:21.502605 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:21.503777 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:21.503788 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:21.504656 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:21.505282 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:21.505301 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:21.507053 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:21.507057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:21.508551 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:21.508564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:21.509733 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:21.509738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:21.509943 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:21.510775 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:21.510805 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:21.513308 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:21.516680 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:21.516689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:21.516706 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:21.517150 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:21.517593 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:21.517599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:21.517603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:21.517801 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:21.518417 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:21.518428 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:21.518941 (+ 513us) client_negotiation.cc:769] Sending connection context
0806 09:10:21.519382 (+ 441us) client_negotiation.cc:240] Negotiation successful
0806 09:10:21.520035 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":395,"threads_started":1}
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
I20260806 09:10:21.523727 3703 heartbeater.cc:347] Connected to a master server at 127.3.69.62:44121
I20260806 09:10:21.524147 3703 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:21.525121 3703 heartbeater.cc:511] Master 127.3.69.62:44121 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
I20260806 09:10:21.529320 3416 ts_manager.cc:194] Registered new tserver with Master: 11abbd3f664e4027882adff6b34c6a86 (127.3.69.1:46857)
I20260806 09:10:21.531500 3416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:44097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:10:21.550143 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:44121
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:21.918933 3713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:21.919301 3713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:21.919469 3713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:21.926803 3713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:21.926941 3713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:21.927187 3713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:10:21.937041 3713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:44121
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:21.938220 3713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:21.939673 3713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:21.953697 3727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:21.953792 3723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:21.953768 3721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:21.954679 3713 server_base.cc:1034] running on GCE node
I20260806 09:10:21.955368 3713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:21.956339 3713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:21.957548 3713 hybrid_clock.cc:648] HybridClock initialized: now 1786007421957499 us; error 36 us; skew 500 ppm
I20260806 09:10:21.957881 3713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:21 dist-test-slave-w4v5 krb5kdc[3356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:21.961303 3713 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:10:21.962798 3713 webserver.cc:460] Webserver started at http://127.3.69.2:38311/ using document root <none> and password file <none>
I20260806 09:10:21.963531 3713 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:21.963661 3713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:21.963994 3713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:21.966971 3713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "3b7e6a5fbdab4a27a07e59f4178cc547"
format_stamp: "Formatted at 2026-08-06 09:10:21 on dist-test-slave-w4v5"
server_key: "0435822f431ae4214ab43b572577e478"
server_key_iv: "56058047f7348a0781db37b18d758a20"
server_key_version: "encryptionkey@0"
I20260806 09:10:21.967801 3713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "3b7e6a5fbdab4a27a07e59f4178cc547"
format_stamp: "Formatted at 2026-08-06 09:10:21 on dist-test-slave-w4v5"
server_key: "0435822f431ae4214ab43b572577e478"
server_key_iv: "56058047f7348a0781db37b18d758a20"
server_key_version: "encryptionkey@0"
I20260806 09:10:21.973052 3713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:10:21.976570 3735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:21.977900 3713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:10:21.978078 3713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "3b7e6a5fbdab4a27a07e59f4178cc547"
format_stamp: "Formatted at 2026-08-06 09:10:21 on dist-test-slave-w4v5"
server_key: "0435822f431ae4214ab43b572577e478"
server_key_iv: "56058047f7348a0781db37b18d758a20"
server_key_version: "encryptionkey@0"
I20260806 09:10:21.978282 3713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:21.994033 3713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:21.998245 3713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:21.998570 3713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:21.999675 3713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:22.001178 3713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:22.001278 3713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:22.001418 3713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:22.001596 3713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:22.042697 3713 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:37821
I20260806 09:10:22.042743 3898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:37821 every 8 connection(s)
I20260806 09:10:22.044118 3713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:10:22.050820 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3713
I20260806 09:10:22.051039 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:10:22.051672 3348 external_mini_cluster.cc:1428] Setting key 2e1fa8056930ce0b609e117d0f5dce52
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:22.070068 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.047035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:51307 (local address 127.3.69.62:44121)
0806 09:10:22.047660 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:22.047666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:22.048730 (+ 1064us) server_negotiation.cc:407] Connection header received
0806 09:10:22.050934 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:22.050939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:22.051033 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:22.051205 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:22.053136 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.054114 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.057147 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.057667 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.057771 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:22.062373 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:22.062383 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:22.062390 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:22.062436 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:22.066349 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.067094 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.067102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.067108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.067295 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.067839 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.067847 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.067851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.068264 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:22.068426 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:22.069202 (+ 776us) server_negotiation.cc:299] Negotiation successful
0806 09:10:22.069467 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":327,"threads_started":1}
I20260806 09:10:22.070624 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.047666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:44121 (local address 127.3.69.2:51307)
0806 09:10:22.048559 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:22.048607 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:22.050606 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:22.051479 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:22.051491 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:22.052230 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:22.052963 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:22.052983 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.054290 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.054298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:22.056961 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:22.057001 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.057947 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.057958 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:22.058350 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:22.059144 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:22.059187 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:22.062119 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:22.066499 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:22.066505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:22.066519 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:22.066931 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:22.067424 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:22.067429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:22.067432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:22.067685 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:22.068440 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:22.068446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:22.068814 (+ 368us) client_negotiation.cc:769] Sending connection context
0806 09:10:22.069053 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 09:10:22.069433 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":330,"threads_started":1}
I20260806 09:10:22.072271 3899 heartbeater.cc:347] Connected to a master server at 127.3.69.62:44121
I20260806 09:10:22.072621 3899 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:22.073408 3899 heartbeater.cc:511] Master 127.3.69.62:44121 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
I20260806 09:10:22.075068 3416 ts_manager.cc:194] Registered new tserver with Master: 3b7e6a5fbdab4a27a07e59f4178cc547 (127.3.69.2:37821)
I20260806 09:10:22.075946 3416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:51307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:10:22.101964 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:44121
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:22.464627 3909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:22.464924 3909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:22.465106 3909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:22.472734 3909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:22.472847 3909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:22.473060 3909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:10:22.483469 3909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:44121
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:22.484779 3909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:22.486166 3909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:22.501428 3919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:22.501413 3917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:22.501470 3916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:22.502146 3909 server_base.cc:1034] running on GCE node
I20260806 09:10:22.502885 3909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:22.503957 3909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:22.505261 3909 hybrid_clock.cc:648] HybridClock initialized: now 1786007422505143 us; error 62 us; skew 500 ppm
I20260806 09:10:22.505620 3909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007422, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:22.509132 3909 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:10:22.510612 3909 webserver.cc:460] Webserver started at http://127.3.69.3:46797/ using document root <none> and password file <none>
I20260806 09:10:22.511344 3909 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:22.511451 3909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:22.511711 3909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:22.514760 3909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "f03760d5fb43451a93c49a9cf5cebdea"
format_stamp: "Formatted at 2026-08-06 09:10:22 on dist-test-slave-w4v5"
server_key: "544c41805d45cbbe9008fb645825baec"
server_key_iv: "b1bd8036356498b58bf02a27320ce3d1"
server_key_version: "encryptionkey@0"
I20260806 09:10:22.515601 3909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "f03760d5fb43451a93c49a9cf5cebdea"
format_stamp: "Formatted at 2026-08-06 09:10:22 on dist-test-slave-w4v5"
server_key: "544c41805d45cbbe9008fb645825baec"
server_key_iv: "b1bd8036356498b58bf02a27320ce3d1"
server_key_version: "encryptionkey@0"
I20260806 09:10:22.520860 3909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:10:22.524413 3930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:22.525890 3909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:10:22.526125 3909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "f03760d5fb43451a93c49a9cf5cebdea"
format_stamp: "Formatted at 2026-08-06 09:10:22 on dist-test-slave-w4v5"
server_key: "544c41805d45cbbe9008fb645825baec"
server_key_iv: "b1bd8036356498b58bf02a27320ce3d1"
server_key_version: "encryptionkey@0"
I20260806 09:10:22.526319 3909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:22.535306 3703 heartbeater.cc:503] Master 127.3.69.62:44121 was elected leader, sending a full tablet report...
I20260806 09:10:22.542548 3909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:22.546989 3909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:22.547345 3909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:22.548403 3909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:22.550045 3909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:22.550143 3909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:22.550292 3909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:22.550367 3909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:22.591529 3909 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:42127
I20260806 09:10:22.591578 4094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:42127 every 8 connection(s)
I20260806 09:10:22.592991 3909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:10:22.601055 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3909
I20260806 09:10:22.601263 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:10:22.601806 3348 external_mini_cluster.cc:1428] Setting key 7e666baa776fe194ba22d14e720f90c6
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007422, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:22.614477 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.595757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:56225 (local address 127.3.69.62:44121)
0806 09:10:22.596332 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:22.596336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:22.597164 (+ 828us) server_negotiation.cc:407] Connection header received
0806 09:10:22.598982 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:22.598986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:22.599064 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:22.599235 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:22.600710 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.602064 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.604185 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.604788 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.604890 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:22.608851 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:22.608855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:22.608858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:22.608891 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:22.611625 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.612353 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.612358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.612360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.612485 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.612866 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.612869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.612871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.613126 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:22.613235 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:22.613955 (+ 720us) server_negotiation.cc:299] Negotiation successful
0806 09:10:22.614132 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":274,"threads_started":1}
I20260806 09:10:22.615501 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.596070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:44121 (local address 127.3.69.3:56225)
0806 09:10:22.596993 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:22.597038 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:22.598741 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:22.599448 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:22.599456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:22.599971 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:22.600536 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:22.600553 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.602244 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.602248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:22.604021 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:22.604034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.605060 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.605068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:22.605437 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:22.606304 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:22.606336 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:22.608666 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:22.611762 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:22.611768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:22.611780 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:22.612244 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:22.612581 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:22.612584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:22.612586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:22.612769 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:22.613232 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:22.613237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:22.613595 (+ 358us) client_negotiation.cc:769] Sending connection context
0806 09:10:22.613821 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 09:10:22.614242 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":443,"threads_started":1}
I20260806 09:10:22.616915 4097 heartbeater.cc:347] Connected to a master server at 127.3.69.62:44121
I20260806 09:10:22.617215 4097 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:22.618189 4097 heartbeater.cc:511] Master 127.3.69.62:44121 requested a full tablet report, sending...
I20260806 09:10:22.619789 3416 ts_manager.cc:194] Registered new tserver with Master: f03760d5fb43451a93c49a9cf5cebdea (127.3.69.3:42127)
I20260806 09:10:22.620591 3416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:56225
I20260806 09:10:22.630999 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:22.682960 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.666766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.3.69.1:46857)
0806 09:10:22.667564 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:22.667571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:22.667600 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:10:22.667667 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:22.667673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:22.667892 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:22.668033 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:22.669830 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.670903 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.672345 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.672908 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.673018 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:22.677056 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:22.677067 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:22.677082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:22.677119 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:22.680064 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.680739 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.680746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.680749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.680877 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.681249 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.681254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.681255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.681610 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:22.681834 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:22.682296 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 09:10:22.682577 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":487,"threads_started":1}
W20260806 09:10:22.684429 3573 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42482
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:22.732451 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.717670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.3.69.1:46857)
0806 09:10:22.717980 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:22.717987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:22.718278 (+ 291us) server_negotiation.cc:407] Connection header received
0806 09:10:22.718500 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:22.718504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:22.718640 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:22.718755 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:22.720177 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.720911 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.722207 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.722632 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.722767 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:22.727192 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:22.727197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:22.727201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:22.727235 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:22.729784 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.730421 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.730428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.730431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.730579 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.730976 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.730982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.730985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.731311 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:22.731563 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:22.731889 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 09:10:22.732119 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260806 09:10:22.734467 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3507
I20260806 09:10:22.747427 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.735526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.3.69.1:46857)
0806 09:10:22.735692 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:22.735696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:22.736173 (+ 477us) server_negotiation.cc:407] Connection header received
0806 09:10:22.736276 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:22.736280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:22.736333 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:22.736410 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:22.737531 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.738204 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.739462 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.739893 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.739989 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:22.742199 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:22.742203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:22.742206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:22.742235 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:22.744853 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.745556 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.745560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.745562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.745688 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.746058 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.746062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.746064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.746325 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:22.746541 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:22.746778 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 09:10:22.746929 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 09:10:22.748205 3696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:22.838893 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3507
I20260806 09:10:22.862305 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3713
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:10:22.878064 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.863259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.3.69.2:37821)
0806 09:10:22.863897 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:22.863904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:22.863931 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:10:22.863994 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:22.864001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:22.864220 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:22.864358 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:22.865494 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.866505 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.867885 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.868471 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.868597 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:22.872680 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:22.872692 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:22.872706 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:22.872741 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:22.875641 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.876232 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.876239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.876241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.876357 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:22.876684 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:22.876687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:22.876689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:22.877049 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:22.877294 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:22.877584 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 09:10:22.877779 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":358,"threads_started":1}
I20260806 09:10:22.879453 3894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:22.964618 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3713
I20260806 09:10:22.989874 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3909
Aug 06 09:10:22 dist-test-slave-w4v5 krb5kdc[3356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:10:23.007392 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:22.990995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.3.69.3:42127)
0806 09:10:22.991767 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:22.991774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:22.991804 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:10:22.991865 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:22.991871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:22.992088 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:22.992222 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:22.993544 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.994636 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.996168 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:22.996763 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:22.996874 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:23.001198 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:23.001210 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:23.001226 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:23.001265 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:23.004417 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:23.005117 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:23.005125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:23.005127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:23.005256 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:23.005670 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:23.005674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:23.005676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:23.006039 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:23.006281 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:23.006689 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 09:10:23.006972 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":435,"threads_started":1}
I20260806 09:10:23.008836 4089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:23.092453 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3909
I20260806 09:10:23.116115 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3373
I20260806 09:10:23.117401 3458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:23.222911 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3373
2026-08-06T09:10:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4488 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:23 dist-test-slave-w4v5 krb5kdc[4124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:23 dist-test-slave-w4v5 krb5kdc[4124](info): set up 2 sockets
Aug 06 09:10:23 dist-test-slave-w4v5 krb5kdc[4124](info): commencing operation
krb5kdc: starting...
W20260806 09:10:24.508018 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:24 dist-test-slave-w4v5 krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:24.702881 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:41561
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:41561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:25.032100 4141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:25.032401 4141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:25.032557 4141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:25.039714 4141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:25.039844 4141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:25.039932 4141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:25.040043 4141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:25.040122 4141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:25.049798 4141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:41561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:41561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:25.051182 4141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:25.052547 4141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:25.064513 4154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:25.064545 4151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:25.065090 4156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:25.065595 4141 server_base.cc:1034] running on GCE node
I20260806 09:10:25.066366 4141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:25.067975 4141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:25.069226 4141 hybrid_clock.cc:648] HybridClock initialized: now 1786007425069182 us; error 48 us; skew 500 ppm
I20260806 09:10:25.069597 4141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:25 dist-test-slave-w4v5 krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:25.073029 4141 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:25.074594 4141 webserver.cc:460] Webserver started at http://127.3.69.62:38135/ using document root <none> and password file <none>
I20260806 09:10:25.075374 4141 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:25.075518 4141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:25.075838 4141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:25.078756 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "dc1848f701004a1b9ac647c5909b3d2c"
format_stamp: "Formatted at 2026-08-06 09:10:25 on dist-test-slave-w4v5"
server_key: "1268c0459e2721eb721e25f894aefa9c"
server_key_iv: "1c63606323ce34882073672750997e20"
server_key_version: "encryptionkey@0"
I20260806 09:10:25.079530 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "dc1848f701004a1b9ac647c5909b3d2c"
format_stamp: "Formatted at 2026-08-06 09:10:25 on dist-test-slave-w4v5"
server_key: "1268c0459e2721eb721e25f894aefa9c"
server_key_iv: "1c63606323ce34882073672750997e20"
server_key_version: "encryptionkey@0"
I20260806 09:10:25.084486 4141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 09:10:25.087903 4166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:25.089179 4141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:10:25.089362 4141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "dc1848f701004a1b9ac647c5909b3d2c"
format_stamp: "Formatted at 2026-08-06 09:10:25 on dist-test-slave-w4v5"
server_key: "1268c0459e2721eb721e25f894aefa9c"
server_key_iv: "1c63606323ce34882073672750997e20"
server_key_version: "encryptionkey@0"
I20260806 09:10:25.089562 4141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:25.114087 4141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:25.118592 4141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:25.118898 4141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:25.137730 4141 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:41561
I20260806 09:10:25.137773 4240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:41561 every 8 connection(s)
I20260806 09:10:25.139506 4141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:25.141139 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4141
I20260806 09:10:25.141371 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:25.142246 3348 external_mini_cluster.cc:1428] Setting key 3842ea6fb40d0bc158340fd2be84d0b6
I20260806 09:10:25.144459 4242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:10:25 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:25.158357 4242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c: Bootstrap starting.
I20260806 09:10:25.161531 4242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:25.162532 4242 log.cc:824] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:25.166100 4242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c: No bootstrap required, opened a new log
I20260806 09:10:25.166152 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:25.144388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.3.69.62:41561)
0806 09:10:25.145256 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:25.145268 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:25.145310 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:10:25.146954 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:25.146974 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:25.147424 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:25.147764 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:25.149658 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:25.151026 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:25.152795 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:25.153420 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:25.153545 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:25.158317 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:25.158325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:25.158338 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:25.158378 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:25.161475 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:25.162171 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:25.162178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:25.162184 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:25.162360 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:25.162763 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:25.162768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:25.162770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:25.163527 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:25.163763 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:25.164386 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 09:10:25.164790 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":450,"threads_started":1}
I20260806 09:10:25.173430 4242 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1848f701004a1b9ac647c5909b3d2c" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41561 } }
I20260806 09:10:25.173758 4242 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:25.173830 4242 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1848f701004a1b9ac647c5909b3d2c, State: Initialized, Role: FOLLOWER
I20260806 09:10:25.174279 4242 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1848f701004a1b9ac647c5909b3d2c" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41561 } }
I20260806 09:10:25.174463 4242 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:25.174568 4242 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:25.174688 4242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:25.176819 4242 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1848f701004a1b9ac647c5909b3d2c" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41561 } }
I20260806 09:10:25.177294 4242 leader_election.cc:302] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1848f701004a1b9ac647c5909b3d2c; no voters:
I20260806 09:10:25.177860 4242 leader_election.cc:288] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:25.178020 4250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:25.178689 4250 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [term 1 LEADER]: Becoming Leader. State: Replica: dc1848f701004a1b9ac647c5909b3d2c, State: Running, Role: LEADER
I20260806 09:10:25.179319 4250 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1848f701004a1b9ac647c5909b3d2c" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41561 } }
I20260806 09:10:25.179881 4242 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:25.182411 4251 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc1848f701004a1b9ac647c5909b3d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1848f701004a1b9ac647c5909b3d2c" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41561 } } }
I20260806 09:10:25.182814 4251 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:25.182339 4253 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1848f701004a1b9ac647c5909b3d2c. Latest consensus state: current_term: 1 leader_uuid: "dc1848f701004a1b9ac647c5909b3d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1848f701004a1b9ac647c5909b3d2c" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41561 } } }
I20260806 09:10:25.183421 4253 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:25.183842 4261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:25.190231 4261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:25.200013 4261 catalog_manager.cc:1347] Generated new cluster ID: 43cfbffb82384dc1b20c3d3338e3583c
I20260806 09:10:25.200145 4261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:25.240571 4261 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:25.242079 4261 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:25.249182 4261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc1848f701004a1b9ac647c5909b3d2c: Generated new TSK 0
I20260806 09:10:25.250375 4261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:10:25.301296 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:41561
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:25.632875 4278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:25.633188 4278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:25.633368 4278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:25.640344 4278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:25.640488 4278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:25.640690 4278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:10:25.650120 4278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:41561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:25.651398 4278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:25.652734 4278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:25.665928 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:25.665969 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:25.666568 4287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:25.667603 4278 server_base.cc:1034] running on GCE node
I20260806 09:10:25.668223 4278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:25.669159 4278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:25.670431 4278 hybrid_clock.cc:648] HybridClock initialized: now 1786007425670389 us; error 59 us; skew 500 ppm
I20260806 09:10:25.670825 4278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:25 dist-test-slave-w4v5 krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:25.674178 4278 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:10:25.675693 4278 webserver.cc:460] Webserver started at http://127.3.69.1:45013/ using document root <none> and password file <none>
I20260806 09:10:25.676373 4278 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:25.676522 4278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:25.676844 4278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:25.679827 4278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "faf408f8ebde4264888a493e02834bfa"
format_stamp: "Formatted at 2026-08-06 09:10:25 on dist-test-slave-w4v5"
server_key: "c48e60d8f057ba1c7df63ba24d20cd6a"
server_key_iv: "8ac77933e6091541222a515abb723b76"
server_key_version: "encryptionkey@0"
I20260806 09:10:25.680691 4278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "faf408f8ebde4264888a493e02834bfa"
format_stamp: "Formatted at 2026-08-06 09:10:25 on dist-test-slave-w4v5"
server_key: "c48e60d8f057ba1c7df63ba24d20cd6a"
server_key_iv: "8ac77933e6091541222a515abb723b76"
server_key_version: "encryptionkey@0"
I20260806 09:10:25.685968 4278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:10:25.689525 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:25.690860 4278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:10:25.691042 4278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "faf408f8ebde4264888a493e02834bfa"
format_stamp: "Formatted at 2026-08-06 09:10:25 on dist-test-slave-w4v5"
server_key: "c48e60d8f057ba1c7df63ba24d20cd6a"
server_key_iv: "8ac77933e6091541222a515abb723b76"
server_key_version: "encryptionkey@0"
I20260806 09:10:25.691320 4278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:25.719225 4278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:25.723651 4278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:25.723953 4278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:25.724996 4278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:25.726370 4278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:25.726480 4278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:25.726641 4278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:25.726717 4278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:25.761416 4278 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:34831
I20260806 09:10:25.761507 4466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:34831 every 8 connection(s)
I20260806 09:10:25.762758 4278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:10:25.771922 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4278
I20260806 09:10:25.772207 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:10:25.773020 3348 external_mini_cluster.cc:1428] Setting key eea44af2da7d903657dc1188670ae740
Aug 06 09:10:25 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:25.784869 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:25.765405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:50379 (local address 127.3.69.62:41561)
0806 09:10:25.765939 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:25.765944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:25.766720 (+ 776us) server_negotiation.cc:407] Connection header received
0806 09:10:25.768643 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:25.768650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:25.768738 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:25.768858 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:25.770335 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:25.771065 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:25.773635 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:25.774199 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:25.774308 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:25.777930 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:25.777938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:25.777947 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:25.777993 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:25.781577 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:25.782531 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:25.782535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:25.782537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:25.782653 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:25.783171 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:25.783174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:25.783176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:25.783435 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:25.783547 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:25.784450 (+ 903us) server_negotiation.cc:299] Negotiation successful
0806 09:10:25.784617 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":319,"threads_started":1}
I20260806 09:10:25.786412 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:25.765856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:41561 (local address 127.3.69.1:50379)
0806 09:10:25.766545 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:25.766592 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:25.768359 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:25.769102 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:25.769110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:25.769601 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:25.770155 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:25.770177 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:25.771296 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:25.771300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:25.773457 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:25.773473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:25.774434 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:25.774441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:25.774734 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:25.775520 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:25.775543 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:25.777688 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:25.781718 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:25.781729 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:25.781749 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:25.782407 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:25.782764 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:25.782770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:25.782776 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:25.783034 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:25.783564 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:25.783574 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:25.784017 (+ 443us) client_negotiation.cc:769] Sending connection context
0806 09:10:25.784342 (+ 325us) client_negotiation.cc:240] Negotiation successful
0806 09:10:25.784920 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":268,"threads_started":1}
I20260806 09:10:25.788100 4467 heartbeater.cc:347] Connected to a master server at 127.3.69.62:41561
I20260806 09:10:25.788481 4467 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:25.789381 4467 heartbeater.cc:511] Master 127.3.69.62:41561 requested a full tablet report, sending...
I20260806 09:10:25.791796 4192 ts_manager.cc:194] Registered new tserver with Master: faf408f8ebde4264888a493e02834bfa (127.3.69.1:34831)
I20260806 09:10:25.793332 4192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:50379
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:10:25.820199 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:41561
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:26.148782 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:26.149120 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:26.149259 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:26.160848 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:26.161000 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:26.161243 4475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:10:26.171011 4475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:41561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:26.172377 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:26.173754 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:26.187112 4484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:26.187139 4485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:26.186996 4488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:26.188683 4475 server_base.cc:1034] running on GCE node
I20260806 09:10:26.189370 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:26.190310 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:26.191584 4475 hybrid_clock.cc:648] HybridClock initialized: now 1786007426191504 us; error 52 us; skew 500 ppm
I20260806 09:10:26.191916 4475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:26 dist-test-slave-w4v5 krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:26.195382 4475 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:10:26.196991 4475 webserver.cc:460] Webserver started at http://127.3.69.2:44201/ using document root <none> and password file <none>
I20260806 09:10:26.197705 4475 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:26.197878 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:26.198200 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:26.201160 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "247c866653424089a1f24f0a3c298a6d"
format_stamp: "Formatted at 2026-08-06 09:10:26 on dist-test-slave-w4v5"
server_key: "069133149a2f5674fee5af5a5add8cc7"
server_key_iv: "cb6ba535bb3de6a9e859c8b735a7038f"
server_key_version: "encryptionkey@0"
I20260806 09:10:26.201953 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "247c866653424089a1f24f0a3c298a6d"
format_stamp: "Formatted at 2026-08-06 09:10:26 on dist-test-slave-w4v5"
server_key: "069133149a2f5674fee5af5a5add8cc7"
server_key_iv: "cb6ba535bb3de6a9e859c8b735a7038f"
server_key_version: "encryptionkey@0"
I20260806 09:10:26.207006 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 09:10:26.210510 4499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:26.211930 4475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:10:26.212102 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "247c866653424089a1f24f0a3c298a6d"
format_stamp: "Formatted at 2026-08-06 09:10:26 on dist-test-slave-w4v5"
server_key: "069133149a2f5674fee5af5a5add8cc7"
server_key_iv: "cb6ba535bb3de6a9e859c8b735a7038f"
server_key_version: "encryptionkey@0"
I20260806 09:10:26.212301 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:26.240945 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:26.244975 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:26.245287 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:26.246279 4475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:26.247713 4475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:26.247814 4475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:26.247953 4475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:26.248025 4475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:26.282390 4475 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:42315
I20260806 09:10:26.282440 4660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:42315 every 8 connection(s)
I20260806 09:10:26.283748 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:10:26.289376 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4475
I20260806 09:10:26.289577 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:10:26.290021 3348 external_mini_cluster.cc:1428] Setting key 2cbb193eb0057c5ed4cf857070f7a6ed
Aug 06 09:10:26 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:26.306564 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.286278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:41103 (local address 127.3.69.62:41561)
0806 09:10:26.286747 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:26.286752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:26.287703 (+ 951us) server_negotiation.cc:407] Connection header received
0806 09:10:26.289733 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:26.289740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:26.289830 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:26.289947 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:26.291629 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.292662 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.295325 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.295812 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.295963 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.300211 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:26.300216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:26.300220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:26.300252 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:26.303454 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:26.304192 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:26.304199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:26.304202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:26.304337 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:26.304785 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:26.304790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:26.304795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:26.305061 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:26.305234 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:26.305945 (+ 711us) server_negotiation.cc:299] Negotiation successful
0806 09:10:26.306170 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260806 09:10:26.307636 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.286717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:41561 (local address 127.3.69.2:41103)
0806 09:10:26.287512 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:26.287587 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:26.289410 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:26.290204 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:26.290212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:26.290732 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:26.291458 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.291482 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.292871 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.292879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.295179 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.295198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.296058 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.296069 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.296401 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.297156 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:26.297189 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:26.299952 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:26.303603 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:26.303612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:26.303629 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:26.304058 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:26.304453 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:26.304459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:26.304462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:26.304658 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:26.305206 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:26.305214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:26.305588 (+ 374us) client_negotiation.cc:769] Sending connection context
0806 09:10:26.305850 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 09:10:26.306422 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":356,"threads_started":1}
I20260806 09:10:26.309271 4662 heartbeater.cc:347] Connected to a master server at 127.3.69.62:41561
I20260806 09:10:26.309630 4662 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
I20260806 09:10:26.310585 4662 heartbeater.cc:511] Master 127.3.69.62:41561 requested a full tablet report, sending...
I20260806 09:10:26.312408 4192 ts_manager.cc:194] Registered new tserver with Master: 247c866653424089a1f24f0a3c298a6d (127.3.69.2:42315)
I20260806 09:10:26.313350 4192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:41103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:10:26.337908 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:41561
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:26.665500 4673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:26.665760 4673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:26.665858 4673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:26.672940 4673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:26.673038 4673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:26.673151 4673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:10:26.682771 4673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:41561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:26.684103 4673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:26.685451 4673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:26.699087 4682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:26.698966 4685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:26.699013 4683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:26.700140 4673 server_base.cc:1034] running on GCE node
I20260806 09:10:26.700762 4673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:26.701704 4673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:26.702912 4673 hybrid_clock.cc:648] HybridClock initialized: now 1786007426702864 us; error 35 us; skew 500 ppm
I20260806 09:10:26.703271 4673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:26 dist-test-slave-w4v5 krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:26.706398 4673 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:10:26.707831 4673 webserver.cc:460] Webserver started at http://127.3.69.3:36763/ using document root <none> and password file <none>
I20260806 09:10:26.708488 4673 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:26.708669 4673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:26.709087 4673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:26.712196 4673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "127323f2243b42d2ae05d8deeb5a1fd9"
format_stamp: "Formatted at 2026-08-06 09:10:26 on dist-test-slave-w4v5"
server_key: "be56f9fc5cab054a741f30845c252fee"
server_key_iv: "be1cb7688c6795ceebe8229965eb0aae"
server_key_version: "encryptionkey@0"
I20260806 09:10:26.713045 4673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "127323f2243b42d2ae05d8deeb5a1fd9"
format_stamp: "Formatted at 2026-08-06 09:10:26 on dist-test-slave-w4v5"
server_key: "be56f9fc5cab054a741f30845c252fee"
server_key_iv: "be1cb7688c6795ceebe8229965eb0aae"
server_key_version: "encryptionkey@0"
I20260806 09:10:26.718479 4673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:10:26.721998 4693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:26.723479 4673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 09:10:26.723726 4673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "127323f2243b42d2ae05d8deeb5a1fd9"
format_stamp: "Formatted at 2026-08-06 09:10:26 on dist-test-slave-w4v5"
server_key: "be56f9fc5cab054a741f30845c252fee"
server_key_iv: "be1cb7688c6795ceebe8229965eb0aae"
server_key_version: "encryptionkey@0"
I20260806 09:10:26.723978 4673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:26.754173 4673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:26.760363 4673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:26.760733 4673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:26.762382 4673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:26.764297 4673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:26.764376 4673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:26.764509 4673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:26.764585 4673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:26.796469 4467 heartbeater.cc:503] Master 127.3.69.62:41561 was elected leader, sending a full tablet report...
I20260806 09:10:26.806649 4673 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:32963
I20260806 09:10:26.806720 4858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:32963 every 8 connection(s)
I20260806 09:10:26.808079 4673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:10:26.816924 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4673
I20260806 09:10:26.817147 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:10:26.817597 3348 external_mini_cluster.cc:1428] Setting key 947cd3d676812f605e351aae760f05c4
Aug 06 09:10:26 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:26.829612 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.810543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:55403 (local address 127.3.69.62:41561)
0806 09:10:26.811026 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:26.811030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:26.811994 (+ 964us) server_negotiation.cc:407] Connection header received
0806 09:10:26.814153 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:26.814159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:26.814261 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:26.814387 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:26.816012 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.817125 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.819224 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.819838 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.819956 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.823836 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:26.823841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:26.823844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:26.823886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:26.826540 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:26.827397 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:26.827401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:26.827403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:26.827594 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:26.827993 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:26.827996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:26.827998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:26.828306 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:26.828438 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:26.829159 (+ 721us) server_negotiation.cc:299] Negotiation successful
0806 09:10:26.829336 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":276,"threads_started":1}
I20260806 09:10:26.830619 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.811023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:41561 (local address 127.3.69.3:55403)
0806 09:10:26.811830 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:26.811873 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:26.813870 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:26.814645 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:26.814653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:26.815223 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:26.815835 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.815852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.817318 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.817322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.818998 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.819016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.820085 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.820092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.820412 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.821260 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:26.821295 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:26.823624 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:26.826679 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:26.826703 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:26.826726 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:26.827279 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:26.827700 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:26.827703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:26.827705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:26.827891 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:26.828451 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:26.828459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:26.828788 (+ 329us) client_negotiation.cc:769] Sending connection context
0806 09:10:26.829011 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 09:10:26.829395 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":282,"threads_started":1}
I20260806 09:10:26.832237 4860 heartbeater.cc:347] Connected to a master server at 127.3.69.62:41561
I20260806 09:10:26.832580 4860 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:26.833474 4860 heartbeater.cc:511] Master 127.3.69.62:41561 requested a full tablet report, sending...
I20260806 09:10:26.834899 4192 ts_manager.cc:194] Registered new tserver with Master: 127323f2243b42d2ae05d8deeb5a1fd9 (127.3.69.3:32963)
I20260806 09:10:26.835759 4192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:55403
I20260806 09:10:26.845631 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:10:26.866510 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.853067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.3.69.62:41561)
0806 09:10:26.853273 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:26.853282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:26.853739 (+ 457us) server_negotiation.cc:407] Connection header received
0806 09:10:26.853939 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:26.853944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:26.854021 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:26.854134 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:26.855801 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.856537 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.857790 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.858216 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.858360 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.860805 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:26.860816 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:26.860820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:26.860853 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:26.863851 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:26.864518 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:26.864530 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:26.864532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:26.864656 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:26.865033 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:26.865038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:26.865042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:26.865359 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:26.865629 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:26.865951 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 09:10:26.866191 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 09:10:26.875384 4192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:10:26.884025 4192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:26.923305 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.900858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34831 (local address 127.0.0.1:57520)
0806 09:10:26.913308 (+ 12450us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:26.913352 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:26.913533 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:26.914934 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:26.914943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:26.914995 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:26.915859 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.915876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.917942 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.917947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.920352 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.920364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.922189 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.922198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.922420 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.922492 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 09:10:26.922644 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 09:10:26.922834 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12181,"mutex_wait_us":392,"spinlock_wait_cycles":7168,"thread_start_us":11991,"threads_started":1}
I20260806 09:10:26.923545 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.901650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.3.69.3:32963)
0806 09:10:26.913500 (+ 11850us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:26.913515 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:26.914302 (+ 787us) server_negotiation.cc:407] Connection header received
0806 09:10:26.914419 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:26.914426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:26.914686 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:26.915764 (+ 1078us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:26.916073 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.917782 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.920926 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.922351 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.922484 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.922719 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:26.922958 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 09:10:26.923167 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10926,"thread_start_us":10687,"threads_started":1}
I20260806 09:10:26.923406 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.901169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.3.69.1:34831)
0806 09:10:26.914286 (+ 13117us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:26.914294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:26.914320 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:10:26.914416 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:26.914424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:26.914677 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:26.915115 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:26.916087 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.917778 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.920513 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.921950 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.922088 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.922342 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:26.922866 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 09:10:26.923040 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12919,"thread_start_us":12759,"threads_started":1}
I20260806 09:10:26.924290 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.901077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:32963 (local address 127.0.0.1:49060)
0806 09:10:26.914023 (+ 12946us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:26.914044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:26.914173 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:26.914934 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:26.914943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:26.914987 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:26.915503 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.915518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.917942 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.917947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.920736 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.920752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.922547 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.922553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.922739 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.922770 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 09:10:26.923879 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0806 09:10:26.924011 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12806,"mutex_wait_us":1135,"spinlock_wait_cycles":768,"thread_start_us":12643,"threads_started":1}
I20260806 09:10:26.928408 4766 tablet_service.cc:1467] Processing CreateTablet for tablet b00741b91ee848ecb5ff6d54f563cdfe (DEFAULT_TABLE table=test-table [id=7d8bf0dd38104f4b89bda4768f4db603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:26.929124 4889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.914378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:42315 (local address 127.0.0.1:39628)
0806 09:10:26.919043 (+ 4665us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:26.919061 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:26.919229 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:26.921799 (+ 2570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:26.921805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:26.921833 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:26.922253 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.922264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.924491 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.924496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.926467 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:26.926486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.928382 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.928386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:26.928502 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.928526 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:10:26.928706 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 09:10:26.928799 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4539,"thread_start_us":4351,"threads_started":1}
I20260806 09:10:26.929157 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:26.914475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.3.69.2:42315)
0806 09:10:26.918515 (+ 4040us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:26.918525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:26.920922 (+ 2397us) server_negotiation.cc:407] Connection header received
0806 09:10:26.921034 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:26.921044 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:26.921335 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:26.921538 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:26.922450 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.924322 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.926702 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:26.928180 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:26.928308 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:26.928500 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:26.928664 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 09:10:26.928829 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3876,"thread_start_us":3761,"threads_started":1}
I20260806 09:10:26.930434 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00741b91ee848ecb5ff6d54f563cdfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:26.929924 4374 tablet_service.cc:1467] Processing CreateTablet for tablet b00741b91ee848ecb5ff6d54f563cdfe (DEFAULT_TABLE table=test-table [id=7d8bf0dd38104f4b89bda4768f4db603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:26.931348 4374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00741b91ee848ecb5ff6d54f563cdfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:26.933198 4570 tablet_service.cc:1467] Processing CreateTablet for tablet b00741b91ee848ecb5ff6d54f563cdfe (DEFAULT_TABLE table=test-table [id=7d8bf0dd38104f4b89bda4768f4db603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:26.934507 4570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00741b91ee848ecb5ff6d54f563cdfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:26.943720 4892 tablet_bootstrap.cc:492] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa: Bootstrap starting.
I20260806 09:10:26.945732 4893 tablet_bootstrap.cc:492] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9: Bootstrap starting.
I20260806 09:10:26.947531 4892 tablet_bootstrap.cc:654] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:26.949014 4892 log.cc:824] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:26.949476 4893 tablet_bootstrap.cc:654] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:26.950769 4894 tablet_bootstrap.cc:492] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d: Bootstrap starting.
I20260806 09:10:26.950937 4893 log.cc:824] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:26.952553 4892 tablet_bootstrap.cc:492] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa: No bootstrap required, opened a new log
I20260806 09:10:26.952837 4892 ts_tablet_manager.cc:1397] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 09:10:26.954331 4894 tablet_bootstrap.cc:654] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:26.954228 4893 tablet_bootstrap.cc:492] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9: No bootstrap required, opened a new log
I20260806 09:10:26.954977 4893 ts_tablet_manager.cc:1397] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 09:10:26.955595 4894 log.cc:824] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:26.958647 4894 tablet_bootstrap.cc:492] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d: No bootstrap required, opened a new log
I20260806 09:10:26.958910 4894 ts_tablet_manager.cc:1397] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 09:10:26.961131 4892 raft_consensus.cc:348] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:26.961511 4892 raft_consensus.cc:374] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:26.961620 4892 raft_consensus.cc:729] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf408f8ebde4264888a493e02834bfa, State: Initialized, Role: FOLLOWER
I20260806 09:10:26.962348 4892 consensus_queue.cc:260] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:26.964350 4892 ts_tablet_manager.cc:1428] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 09:10:26.965778 4893 raft_consensus.cc:348] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:26.966225 4893 raft_consensus.cc:374] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:26.966336 4893 raft_consensus.cc:729] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127323f2243b42d2ae05d8deeb5a1fd9, State: Initialized, Role: FOLLOWER
I20260806 09:10:26.966948 4893 consensus_queue.cc:260] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:26.970942 4894 raft_consensus.cc:348] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:26.971494 4894 raft_consensus.cc:374] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:26.971601 4894 raft_consensus.cc:729] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247c866653424089a1f24f0a3c298a6d, State: Initialized, Role: FOLLOWER
I20260806 09:10:26.971793 4860 heartbeater.cc:503] Master 127.3.69.62:41561 was elected leader, sending a full tablet report...
I20260806 09:10:26.972321 4894 consensus_queue.cc:260] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:26.972631 4893 ts_tablet_manager.cc:1428] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260806 09:10:26.973557 4662 heartbeater.cc:503] Master 127.3.69.62:41561 was elected leader, sending a full tablet report...
I20260806 09:10:26.974231 4894 ts_tablet_manager.cc:1428] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260806 09:10:27.015347 4468 tablet.cc:2369] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:10:27.023861 4898 raft_consensus.cc:482] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:27.024199 4898 raft_consensus.cc:504] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:27.026332 4898 leader_election.cc:288] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 127323f2243b42d2ae05d8deeb5a1fd9 (127.3.69.3:32963), 247c866653424089a1f24f0a3c298a6d (127.3.69.2:42315)
I20260806 09:10:27.033609 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.026841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:33099 (local address 127.3.69.3:32963)
0806 09:10:27.027119 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:27.027126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:27.027556 (+ 430us) server_negotiation.cc:407] Connection header received
0806 09:10:27.028251 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:27.028258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:27.028332 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:27.028435 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:27.029267 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.029992 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.031895 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.032673 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.032834 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.032948 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:27.033225 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 09:10:27.033351 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:10:27.033792 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.026679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:32963 (local address 127.3.69.1:33099)
0806 09:10:27.027440 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:27.027460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:27.028028 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:27.028455 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:27.028459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:27.028497 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:27.029115 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:27.029129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.030126 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.030133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:27.031730 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:27.031743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.032892 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.032898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:27.033062 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.033087 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:10:27.033195 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 09:10:27.033324 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"spinlock_wait_cycles":32640,"thread_start_us":394,"threads_started":1}
I20260806 09:10:27.034379 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.027689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:40697 (local address 127.3.69.2:42315)
0806 09:10:27.027967 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:27.027975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:27.028538 (+ 563us) server_negotiation.cc:407] Connection header received
0806 09:10:27.028874 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:27.028878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:27.028944 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:27.029065 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:27.029700 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.030529 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.032456 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.033337 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.033510 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.033633 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:27.034032 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 09:10:27.034177 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:10:27.034500 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.027531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:42315 (local address 127.3.69.1:40697)
0806 09:10:27.028424 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:27.028445 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:27.028682 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:27.029095 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:27.029099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:27.029127 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:27.029534 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:27.029545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.030702 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.030708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:27.032287 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:27.032301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.033586 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.033593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:27.033833 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.033867 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:10:27.033967 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 09:10:27.034121 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":395,"threads_started":1}
I20260806 09:10:27.035216 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00741b91ee848ecb5ff6d54f563cdfe" candidate_uuid: "faf408f8ebde4264888a493e02834bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" is_pre_election: true
I20260806 09:10:27.035573 4602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00741b91ee848ecb5ff6d54f563cdfe" candidate_uuid: "faf408f8ebde4264888a493e02834bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247c866653424089a1f24f0a3c298a6d" is_pre_election: true
W20260806 09:10:27.035950 4664 tablet.cc:2369] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:10:27.036078 4794 raft_consensus.cc:2454] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faf408f8ebde4264888a493e02834bfa in term 0.
I20260806 09:10:27.036194 4602 raft_consensus.cc:2454] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faf408f8ebde4264888a493e02834bfa in term 0.
I20260806 09:10:27.037034 4300 leader_election.cc:302] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247c866653424089a1f24f0a3c298a6d, faf408f8ebde4264888a493e02834bfa; no voters:
I20260806 09:10:27.037456 4898 raft_consensus.cc:2781] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:27.037649 4898 raft_consensus.cc:482] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:27.037782 4898 raft_consensus.cc:3037] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:27.040373 4898 raft_consensus.cc:504] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:27.041062 4898 leader_election.cc:288] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [CANDIDATE]: Term 1 election: Requested vote from peers 127323f2243b42d2ae05d8deeb5a1fd9 (127.3.69.3:32963), 247c866653424089a1f24f0a3c298a6d (127.3.69.2:42315)
I20260806 09:10:27.041531 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00741b91ee848ecb5ff6d54f563cdfe" candidate_uuid: "faf408f8ebde4264888a493e02834bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "127323f2243b42d2ae05d8deeb5a1fd9"
I20260806 09:10:27.041731 4794 raft_consensus.cc:3037] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:27.041723 4602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00741b91ee848ecb5ff6d54f563cdfe" candidate_uuid: "faf408f8ebde4264888a493e02834bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247c866653424089a1f24f0a3c298a6d"
I20260806 09:10:27.041954 4602 raft_consensus.cc:3037] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:27.044193 4794 raft_consensus.cc:2454] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faf408f8ebde4264888a493e02834bfa in term 1.
I20260806 09:10:27.044440 4602 raft_consensus.cc:2454] T b00741b91ee848ecb5ff6d54f563cdfe P 247c866653424089a1f24f0a3c298a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faf408f8ebde4264888a493e02834bfa in term 1.
I20260806 09:10:27.044608 4300 leader_election.cc:302] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 127323f2243b42d2ae05d8deeb5a1fd9, faf408f8ebde4264888a493e02834bfa; no voters:
I20260806 09:10:27.044915 4898 raft_consensus.cc:2781] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:27.045378 4898 raft_consensus.cc:686] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [term 1 LEADER]: Becoming Leader. State: Replica: faf408f8ebde4264888a493e02834bfa, State: Running, Role: LEADER
I20260806 09:10:27.045877 4898 consensus_queue.cc:237] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } }
I20260806 09:10:27.052065 4191 catalog_manager.cc:5495] T b00741b91ee848ecb5ff6d54f563cdfe P faf408f8ebde4264888a493e02834bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to faf408f8ebde4264888a493e02834bfa (127.3.69.1). New cstate: current_term: 1 leader_uuid: "faf408f8ebde4264888a493e02834bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faf408f8ebde4264888a493e02834bfa" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "127323f2243b42d2ae05d8deeb5a1fd9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247c866653424089a1f24f0a3c298a6d" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42315 } health_report { overall_health: UNKNOWN } } }
W20260806 09:10:27.059851 4862 tablet.cc:2369] T b00741b91ee848ecb5ff6d54f563cdfe P 127323f2243b42d2ae05d8deeb5a1fd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:27.076089 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.060847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.3.69.1:34831)
0806 09:10:27.061018 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:27.061022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:27.061618 (+ 596us) server_negotiation.cc:407] Connection header received
0806 09:10:27.061747 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:27.061750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:27.061814 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:27.061897 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:27.063639 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.064390 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.065624 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.066031 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.066125 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.070511 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:27.070524 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:27.070580 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:27.070614 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:27.073637 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.074238 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.074245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.074247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.074404 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.074745 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.074748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.074750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.075165 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:27.075395 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:27.075648 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 09:10:27.075837 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:27.131373 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.116360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.3.69.1:34831)
0806 09:10:27.116641 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:27.116648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:27.117038 (+ 390us) server_negotiation.cc:407] Connection header received
0806 09:10:27.117270 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:27.117274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:27.117339 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:27.117452 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:27.118889 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.119687 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.121065 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.121491 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.121633 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.125819 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:27.125824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:27.125827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:27.125861 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:27.128637 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.129280 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.129286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.129289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.129418 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.129829 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.129834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.129837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.130129 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:27.130430 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:27.130701 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 09:10:27.130928 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:27.184366 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.169968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.3.69.1:34831)
0806 09:10:27.170232 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:27.170239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:27.170511 (+ 272us) server_negotiation.cc:407] Connection header received
0806 09:10:27.170656 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:27.170660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:27.170745 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:27.170847 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:27.172321 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.173140 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.174300 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.174740 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.174847 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.178711 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:27.178716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:27.178719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:27.178753 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:27.181622 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.182283 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.182293 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.182295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.182474 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.182854 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.182859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.182862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.183216 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:27.183502 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:27.183783 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 09:10:27.183995 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 09:10:27.186559 4343 tablet_service.cc:2908] Rejecting scan request for tablet b00741b91ee848ecb5ff6d54f563cdfe: Uninitialized: clean time has not yet been initialized
I20260806 09:10:27.190636 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4278
I20260806 09:10:27.203728 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.191621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.3.69.1:34831)
0806 09:10:27.191788 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:27.191792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:27.192187 (+ 395us) server_negotiation.cc:407] Connection header received
0806 09:10:27.192332 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:27.192336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:27.192422 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:27.192529 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:27.194173 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.195060 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.196351 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.196752 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.196848 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.198856 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:27.198861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:27.198864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:27.198902 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:27.201336 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.201971 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.201975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.201978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.202115 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.202452 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.202456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.202458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.202703 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:27.202895 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:27.203263 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 09:10:27.203471 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 09:10:27.204370 4459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:27.303452 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4278
I20260806 09:10:27.329897 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4475
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:10:27.345566 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.330717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.3.69.2:42315)
0806 09:10:27.330941 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:27.330945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:27.330969 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:10:27.331262 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:27.331269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:27.331406 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:27.331518 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:27.332733 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.333500 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.334819 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.335478 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.335592 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.340083 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:27.340097 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:27.340113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:27.340156 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:27.343050 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.343694 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.343701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.343703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.343831 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.344163 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.344167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.344169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.344564 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:27.344806 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:27.345033 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 09:10:27.345190 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:10:27.346236 4655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:27.436080 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4475
I20260806 09:10:27.460071 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4673
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:10:27.475876 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:27.461025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.3.69.3:32963)
0806 09:10:27.461236 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:27.461241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:27.461262 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:10:27.461349 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:27.461354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:27.461423 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:27.461515 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:27.462832 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.463864 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.465088 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:27.465461 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:27.465561 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:27.469992 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:27.470020 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:27.470038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:27.470084 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:27.473161 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.473919 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.473927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.473929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.474059 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:27.474410 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:27.474414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:27.474415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:27.474800 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:27.475011 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:27.475315 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 09:10:27.475477 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:10:27.476553 4853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:27.564800 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4673
I20260806 09:10:27.591225 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4141
I20260806 09:10:27.592403 4235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:27.700599 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4141
2026-08-06T09:10:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4477 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4928](info): set up 2 sockets
Aug 06 09:10:27 dist-test-slave-w4v5 krb5kdc[4928](info): commencing operation
krb5kdc: starting...
W20260806 09:10:28.986240 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:29 dist-test-slave-w4v5 krb5kdc[4928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:29.188207 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:34461
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:34461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:29.519872 4949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:29.520212 4949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:29.520381 4949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:29.527572 4949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:29.527719 4949 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:10:29.527817 4949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:29.527943 4949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:29.528033 4949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:29.528126 4949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:29.537948 4949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:34461
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:34461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:29.539207 4949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:29.540511 4949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:29.551990 4957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:29.552557 4959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:29.552018 4961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:29.553237 4949 server_base.cc:1034] running on GCE node
I20260806 09:10:29.554023 4949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:29.555415 4949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:29.556764 4949 hybrid_clock.cc:648] HybridClock initialized: now 1786007429556693 us; error 63 us; skew 500 ppm
I20260806 09:10:29.557086 4949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:29 dist-test-slave-w4v5 krb5kdc[4928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007429, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:29.560184 4949 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:29.561614 4949 webserver.cc:460] Webserver started at http://127.3.69.62:40681/ using document root <none> and password file <none>
I20260806 09:10:29.562291 4949 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:29.562427 4949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:29.562788 4949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:29.565726 4949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "c2ca53879962408a91c4a5b4794d85f8"
format_stamp: "Formatted at 2026-08-06 09:10:29 on dist-test-slave-w4v5"
server_key: "dbf18e45b88e146801f33306f3883731"
server_key_iv: "231c3fa696e0093be58df37f6af73e09"
server_key_version: "encryptionkey@0"
I20260806 09:10:29.566545 4949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "c2ca53879962408a91c4a5b4794d85f8"
format_stamp: "Formatted at 2026-08-06 09:10:29 on dist-test-slave-w4v5"
server_key: "dbf18e45b88e146801f33306f3883731"
server_key_iv: "231c3fa696e0093be58df37f6af73e09"
server_key_version: "encryptionkey@0"
I20260806 09:10:29.571772 4949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 09:10:29.575132 4972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:29.576407 4949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:10:29.576596 4949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "c2ca53879962408a91c4a5b4794d85f8"
format_stamp: "Formatted at 2026-08-06 09:10:29 on dist-test-slave-w4v5"
server_key: "dbf18e45b88e146801f33306f3883731"
server_key_iv: "231c3fa696e0093be58df37f6af73e09"
server_key_version: "encryptionkey@0"
I20260806 09:10:29.576812 4949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:29.598203 4949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:29.602137 4949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:29.602409 4949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:29.624619 4949 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:34461
I20260806 09:10:29.624647 5057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:34461 every 8 connection(s)
I20260806 09:10:29.625952 4949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:29.628257 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4949
I20260806 09:10:29.628525 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:29.629376 3348 external_mini_cluster.cc:1428] Setting key f1dba46f92a43e422bd9192cd9a21d1b
I20260806 09:10:29.630525 5058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:29.642081 5058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8: Bootstrap starting.
Aug 06 09:10:29 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:29.645541 5058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:29.646675 5058 log.cc:824] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:29.649961 5058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8: No bootstrap required, opened a new log
I20260806 09:10:29.651664 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:29.631444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.3.69.62:34461)
0806 09:10:29.632223 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:29.632235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:29.632274 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:10:29.633815 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:29.633836 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:29.634215 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:29.634628 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:29.636135 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:29.637458 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:29.638699 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:29.639466 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:29.639695 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:29.644054 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:29.644065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:29.644078 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:29.644122 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:29.647528 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:29.648281 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:29.648288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:29.648294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:29.648452 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:29.648872 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:29.648875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:29.648877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:29.649389 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:29.649619 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:29.650146 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 09:10:29.650634 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":376,"threads_started":1}
I20260806 09:10:29.657612 5058 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ca53879962408a91c4a5b4794d85f8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34461 } }
I20260806 09:10:29.657956 5058 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:29.658071 5058 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ca53879962408a91c4a5b4794d85f8, State: Initialized, Role: FOLLOWER
I20260806 09:10:29.658612 5058 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ca53879962408a91c4a5b4794d85f8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34461 } }
I20260806 09:10:29.658840 5058 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:29.658996 5058 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:29.659224 5058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:29.661306 5058 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ca53879962408a91c4a5b4794d85f8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34461 } }
I20260806 09:10:29.661826 5058 leader_election.cc:302] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2ca53879962408a91c4a5b4794d85f8; no voters:
I20260806 09:10:29.662375 5058 leader_election.cc:288] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:29.662503 5064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:29.663277 5064 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [term 1 LEADER]: Becoming Leader. State: Replica: c2ca53879962408a91c4a5b4794d85f8, State: Running, Role: LEADER
I20260806 09:10:29.663846 5064 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ca53879962408a91c4a5b4794d85f8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34461 } }
I20260806 09:10:29.664458 5058 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:29.666682 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ca53879962408a91c4a5b4794d85f8. Latest consensus state: current_term: 1 leader_uuid: "c2ca53879962408a91c4a5b4794d85f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ca53879962408a91c4a5b4794d85f8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34461 } } }
I20260806 09:10:29.666922 5068 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2ca53879962408a91c4a5b4794d85f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ca53879962408a91c4a5b4794d85f8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34461 } } }
I20260806 09:10:29.667274 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:29.667371 5068 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:29.671810 5077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:29.676548 5077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:29.684641 5077 catalog_manager.cc:1347] Generated new cluster ID: 18c3e10f447e49df9e55422903a96efc
I20260806 09:10:29.684804 5077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:29.708868 5077 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:29.710167 5077 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:29.720662 5077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8: Generated new TSK 0
I20260806 09:10:29.721526 5077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:10:29.774653 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34461
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:30.105736 5096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:30.106025 5096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:30.106210 5096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:30.113448 5096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:30.113557 5096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:30.113740 5096 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:10:30.113873 5096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:10:30.124035 5096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:30.125360 5096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:30.126735 5096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:30.140081 5104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:30.140156 5106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:30.140183 5110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:30.141220 5096 server_base.cc:1034] running on GCE node
I20260806 09:10:30.141847 5096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:30.142782 5096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:30.144026 5096 hybrid_clock.cc:648] HybridClock initialized: now 1786007430143933 us; error 70 us; skew 500 ppm
I20260806 09:10:30.144335 5096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:30 dist-test-slave-w4v5 krb5kdc[4928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007430, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:30.147655 5096 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:10:30.149116 5096 webserver.cc:460] Webserver started at http://127.3.69.1:41299/ using document root <none> and password file <none>
I20260806 09:10:30.149868 5096 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:30.150000 5096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:30.150344 5096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:30.153712 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "3ec666f619064437b4a2949984067f1d"
format_stamp: "Formatted at 2026-08-06 09:10:30 on dist-test-slave-w4v5"
server_key: "7b66ffc886d9d92adf13bd87fe4a9f03"
server_key_iv: "a00c35ea7a319cf767d1ed955b45e369"
server_key_version: "encryptionkey@0"
I20260806 09:10:30.154518 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec666f619064437b4a2949984067f1d"
format_stamp: "Formatted at 2026-08-06 09:10:30 on dist-test-slave-w4v5"
server_key: "7b66ffc886d9d92adf13bd87fe4a9f03"
server_key_iv: "a00c35ea7a319cf767d1ed955b45e369"
server_key_version: "encryptionkey@0"
I20260806 09:10:30.159842 5096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 09:10:30.163635 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:30.165025 5096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:10:30.165215 5096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "3ec666f619064437b4a2949984067f1d"
format_stamp: "Formatted at 2026-08-06 09:10:30 on dist-test-slave-w4v5"
server_key: "7b66ffc886d9d92adf13bd87fe4a9f03"
server_key_iv: "a00c35ea7a319cf767d1ed955b45e369"
server_key_version: "encryptionkey@0"
I20260806 09:10:30.165444 5096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:30.178308 5096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:30.182598 5096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:30.182857 5096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:30.185521 5096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:30.185607 5096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:30.185750 5096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:30.185804 5096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:10:30 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007430, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:30.211453 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:30.186572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:57299 (local address 127.3.69.62:34461)
0806 09:10:30.187111 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:30.187120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:30.188318 (+ 1198us) server_negotiation.cc:407] Connection header received
0806 09:10:30.190607 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:30.190614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:30.190724 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:30.190858 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:30.193312 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.194630 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.197680 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.198393 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.198574 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:30.204390 (+ 5816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:30.204395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:30.204398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:30.204433 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:30.207373 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:30.208190 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:30.208196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:30.208199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:30.208383 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:30.209083 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:30.209097 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:30.209103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:30.209455 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:30.209646 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:30.210772 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0806 09:10:30.211021 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":276,"threads_started":1}
I20260806 09:10:30.212589 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:30.187223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34461 (local address 127.3.69.1:57299)
0806 09:10:30.188115 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:30.188166 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:30.190243 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:30.191149 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:30.191164 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:30.192160 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:30.193060 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:30.193112 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.194856 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.194869 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:30.197492 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:30.197505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.198687 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.198695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:30.198965 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:30.200033 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:30.200056 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:30.204106 (+ 4050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:30.207557 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:30.207564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:30.207577 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:30.208046 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:30.208524 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:30.208531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:30.208538 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:30.208907 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:30.209745 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:30.209759 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:30.210259 (+ 500us) client_negotiation.cc:769] Sending connection context
0806 09:10:30.210671 (+ 412us) client_negotiation.cc:240] Negotiation successful
0806 09:10:30.211289 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":401,"threads_started":1}
I20260806 09:10:30.235924 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:30.219342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:49601 (local address 127.3.69.62:34461)
0806 09:10:30.219590 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:30.219595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:30.220733 (+ 1138us) server_negotiation.cc:407] Connection header received
0806 09:10:30.221015 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:30.221025 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:30.221125 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:30.221240 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:30.222846 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.223868 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.225917 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.226709 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.226861 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:30.228740 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:30.228749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:30.228756 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:30.228808 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:30.232752 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:30.233635 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:30.233642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:30.233647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:30.233808 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:30.234183 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:30.234192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:30.234196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:30.234603 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:30.234777 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:30.235335 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 09:10:30.235565 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:10:30.236012 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:30.219793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34461 (local address 127.3.69.1:49601)
0806 09:10:30.220580 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:30.220609 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:30.220784 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:30.221310 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:30.221323 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:30.222093 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:30.222624 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:30.222642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.224023 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.224029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:30.225743 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:30.225763 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.226918 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.226923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:30.227036 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:30.227774 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:30.227790 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:30.228515 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:30.232921 (+ 4406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:30.232928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:30.232935 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:30.233512 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:30.233933 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:30.233937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:30.233939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:30.234068 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:30.234778 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:30.234785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:30.235001 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 09:10:30.235221 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 09:10:30.235562 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":475,"threads_started":1}
I20260806 09:10:30.245811 5096 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:34121
I20260806 09:10:30.245831 5303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:34121 every 8 connection(s)
I20260806 09:10:30.247048 5096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:10:30.249133 5304 heartbeater.cc:347] Connected to a master server at 127.3.69.62:34461
I20260806 09:10:30.249377 5304 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:30.250077 5304 heartbeater.cc:511] Master 127.3.69.62:34461 requested a full tablet report, sending...
I20260806 09:10:30.252022 5000 ts_manager.cc:194] Registered new tserver with Master: 3ec666f619064437b4a2949984067f1d (127.3.69.1:34121)
I20260806 09:10:30.253518 5000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:57299
I20260806 09:10:30.254194 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 5096
I20260806 09:10:30.254432 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:10:30.255117 3348 external_mini_cluster.cc:1428] Setting key 514cd5e2acf3f300f53997add460b529
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:10:30.300354 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34461
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:30.633200 5310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:30.633488 5310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:30.633656 5310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:30.640650 5310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:30.640789 5310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:30.640928 5310 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:10:30.641062 5310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:10:30.650311 5310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:30.651677 5310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:30.652980 5310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:30.674216 5318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:30.674233 5320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:30.674796 5317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:30.675457 5310 server_base.cc:1034] running on GCE node
I20260806 09:10:30.676115 5310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:30.676996 5310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:30.678231 5310 hybrid_clock.cc:648] HybridClock initialized: now 1786007430678155 us; error 54 us; skew 500 ppm
I20260806 09:10:30.678539 5310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:30 dist-test-slave-w4v5 krb5kdc[4928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007430, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:30.682129 5310 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:10:30.683766 5310 webserver.cc:460] Webserver started at http://127.3.69.2:37735/ using document root <none> and password file <none>
I20260806 09:10:30.684499 5310 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:30.684626 5310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:30.684980 5310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:30.688017 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "3774798ac00a4a25a537dcbee22aed8c"
format_stamp: "Formatted at 2026-08-06 09:10:30 on dist-test-slave-w4v5"
server_key: "144870b2159c79507a66a754859dcd18"
server_key_iv: "fa7f3890b904b5447054d5701eb89aa3"
server_key_version: "encryptionkey@0"
I20260806 09:10:30.688903 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "3774798ac00a4a25a537dcbee22aed8c"
format_stamp: "Formatted at 2026-08-06 09:10:30 on dist-test-slave-w4v5"
server_key: "144870b2159c79507a66a754859dcd18"
server_key_iv: "fa7f3890b904b5447054d5701eb89aa3"
server_key_version: "encryptionkey@0"
I20260806 09:10:30.694073 5310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:10:30.697448 5328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:30.698828 5310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:10:30.699007 5310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "3774798ac00a4a25a537dcbee22aed8c"
format_stamp: "Formatted at 2026-08-06 09:10:30 on dist-test-slave-w4v5"
server_key: "144870b2159c79507a66a754859dcd18"
server_key_iv: "fa7f3890b904b5447054d5701eb89aa3"
server_key_version: "encryptionkey@0"
I20260806 09:10:30.699290 5310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:30.722236 5310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:30.726203 5310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:30.726486 5310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:30.729177 5310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:30.729288 5310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:30.729449 5310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:30.729527 5310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:10:30 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007430, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:30.755882 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:30.730166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:43375 (local address 127.3.69.62:34461)
0806 09:10:30.730386 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:30.730393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:30.731917 (+ 1524us) server_negotiation.cc:407] Connection header received
0806 09:10:30.734080 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:30.734089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:30.734205 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:30.734291 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:30.737087 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.738192 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.740765 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.741617 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.741792 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:30.747675 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:30.747681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:30.747685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:30.747746 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:30.751401 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:30.752727 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:30.752735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:30.752741 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:30.752885 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:30.753602 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:30.753608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:30.753614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:30.754007 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:30.754215 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:30.755258 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0806 09:10:30.755513 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:10:30.757344 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:30.730742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34461 (local address 127.3.69.2:43375)
0806 09:10:30.731712 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:30.731769 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:30.733720 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:30.734885 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:30.734899 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:30.735838 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:30.736896 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:30.736920 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.738352 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.738358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:30.740557 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:30.740570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.741923 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.741932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:30.742215 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:30.743155 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:30.743201 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:30.747325 (+ 4124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:30.751604 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:30.751627 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:30.751655 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:30.752559 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:30.753015 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:30.753023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:30.753030 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:30.753438 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:30.754155 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:30.754163 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:30.754718 (+ 555us) client_negotiation.cc:769] Sending connection context
0806 09:10:30.755149 (+ 431us) client_negotiation.cc:240] Negotiation successful
0806 09:10:30.755778 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":440,"threads_started":1}
I20260806 09:10:30.786367 5310 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:44639
I20260806 09:10:30.786418 5499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:44639 every 8 connection(s)
I20260806 09:10:30.787951 5310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:10:30.790699 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 5310
I20260806 09:10:30.790904 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:10:30.790462 5500 heartbeater.cc:347] Connected to a master server at 127.3.69.62:34461
I20260806 09:10:30.791517 3348 external_mini_cluster.cc:1428] Setting key 3e625a983fb6537a504c8d7eafb7e732
I20260806 09:10:30.791563 5500 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:30.791962 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:30.772049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:33263 (local address 127.3.69.62:34461)
0806 09:10:30.772285 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:30.772290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:30.773606 (+ 1316us) server_negotiation.cc:407] Connection header received
0806 09:10:30.773897 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:30.773906 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:30.774017 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:30.774166 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:30.776073 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.776931 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.779846 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.780491 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.780607 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:30.783940 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:30.783947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:30.783954 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:30.784001 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:30.787488 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:30.788425 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:30.788430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:30.788432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:30.788677 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:30.789249 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:30.789253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:30.789256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:30.789780 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:30.789975 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:30.791401 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0806 09:10:30.791651 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:10:30.792356 5500 heartbeater.cc:511] Master 127.3.69.62:34461 requested a full tablet report, sending...
I20260806 09:10:30.791183 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:30.772313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34461 (local address 127.3.69.2:33263)
0806 09:10:30.773467 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:30.773486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:30.773703 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:30.774266 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:30.774275 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:30.775229 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:30.775882 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:30.775901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.777125 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.777133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:30.779568 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:30.779587 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:30.780724 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:30.780734 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:30.780980 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:30.781953 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:30.781985 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:30.783667 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:30.787638 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:30.787643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:30.787647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:30.788289 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:30.788842 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:30.788851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:30.788857 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:30.789070 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:30.789928 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:30.789936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:30.790174 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 09:10:30.790338 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 09:10:30.790701 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":700,"threads_started":1}
I20260806 09:10:30.793828 5000 ts_manager.cc:194] Registered new tserver with Master: 3774798ac00a4a25a537dcbee22aed8c (127.3.69.2:44639)
I20260806 09:10:30.794975 5000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:43375
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:10:30.838650 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34461
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:31.170897 5506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:31.171190 5506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:31.171386 5506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:31.178321 5506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:31.178421 5506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:31.178589 5506 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:10:31.178718 5506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:10:31.188438 5506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:34461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:31.189694 5506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:31.191046 5506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:31.204860 5513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:31.204830 5516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:31.204892 5519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:31.205950 5506 server_base.cc:1034] running on GCE node
I20260806 09:10:31.206543 5506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:31.207471 5506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:31.208684 5506 hybrid_clock.cc:648] HybridClock initialized: now 1786007431208625 us; error 29 us; skew 500 ppm
I20260806 09:10:31.208984 5506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:31 dist-test-slave-w4v5 krb5kdc[4928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007431, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:31.212102 5506 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:10:31.213542 5506 webserver.cc:460] Webserver started at http://127.3.69.3:44863/ using document root <none> and password file <none>
I20260806 09:10:31.214222 5506 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:31.214363 5506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:31.214756 5506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:31.217772 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "a3e126dcd72b426991ff8f6cf2193794"
format_stamp: "Formatted at 2026-08-06 09:10:31 on dist-test-slave-w4v5"
server_key: "4563716af13d1e4641ef8e1077467ed2"
server_key_iv: "fa14e4ba13ee0415b5e12ebe3ca129fe"
server_key_version: "encryptionkey@0"
I20260806 09:10:31.218560 5506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "a3e126dcd72b426991ff8f6cf2193794"
format_stamp: "Formatted at 2026-08-06 09:10:31 on dist-test-slave-w4v5"
server_key: "4563716af13d1e4641ef8e1077467ed2"
server_key_iv: "fa14e4ba13ee0415b5e12ebe3ca129fe"
server_key_version: "encryptionkey@0"
I20260806 09:10:31.223798 5506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:10:31.227371 5526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:31.228623 5506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:10:31.228796 5506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "a3e126dcd72b426991ff8f6cf2193794"
format_stamp: "Formatted at 2026-08-06 09:10:31 on dist-test-slave-w4v5"
server_key: "4563716af13d1e4641ef8e1077467ed2"
server_key_iv: "fa14e4ba13ee0415b5e12ebe3ca129fe"
server_key_version: "encryptionkey@0"
I20260806 09:10:31.229022 5506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:31.241816 5506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:31.245673 5506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:31.245944 5506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:31.248812 5506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:31.248965 5506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 09:10:31.249106 5506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:31.249190 5506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:31.256747 5304 heartbeater.cc:503] Master 127.3.69.62:34461 was elected leader, sending a full tablet report...
Aug 06 09:10:31 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007431, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:31.271874 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.249571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:57207 (local address 127.3.69.62:34461)
0806 09:10:31.249776 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.249782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.251503 (+ 1721us) server_negotiation.cc:407] Connection header received
0806 09:10:31.253744 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.253749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.253837 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.253959 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:31.256203 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.257292 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.259323 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.259941 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.260076 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.263991 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:31.263996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:31.263999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:31.264031 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:31.268252 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:31.269391 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:31.269397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:31.269400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:31.269537 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:31.270005 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:31.270010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:31.270014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:31.270356 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:31.270535 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.271269 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.271498 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:10:31.272936 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.250336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34461 (local address 127.3.69.3:57207)
0806 09:10:31.251355 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.251392 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.253429 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.254197 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.254205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.254970 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:31.255975 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.256014 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.257479 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.257484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.259132 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.259148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.260197 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.260205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.260463 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.261330 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:31.261363 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:31.263770 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:31.268419 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:31.268431 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:31.268453 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:31.269219 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:31.269663 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:31.269668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:31.269672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:31.269863 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:31.270506 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:31.270514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:31.270869 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.271170 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.271751 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":398,"threads_started":1}
I20260806 09:10:31.294780 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.279611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:33921 (local address 127.3.69.62:34461)
0806 09:10:31.279814 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.279821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.280943 (+ 1122us) server_negotiation.cc:407] Connection header received
0806 09:10:31.281159 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.281165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.281241 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.281343 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:31.282283 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.283000 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.284951 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.285370 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.285515 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.287328 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:31.287334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:31.287339 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:31.287381 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:31.291502 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:31.292354 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:31.292363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:31.292368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:31.292567 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:31.293050 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:31.293058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:31.293064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:31.293466 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:31.293638 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.294142 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.294384 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 09:10:31.294850 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.279876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34461 (local address 127.3.69.3:33921)
0806 09:10:31.280840 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.280856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.281032 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.281369 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.281373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.281858 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:31.282138 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.282148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.283176 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.283184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.284762 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.284778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.285568 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.285574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.285754 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.286385 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:31.286402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:31.287148 (+ 746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:31.291645 (+ 4497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:31.291650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:31.291654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:31.292208 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:31.292727 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:31.292738 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:31.292742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:31.292909 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:31.293605 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:31.293610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:31.293848 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.294013 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.294400 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":632,"threads_started":1}
I20260806 09:10:31.311524 5506 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:41633
I20260806 09:10:31.311545 5691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:41633 every 8 connection(s)
I20260806 09:10:31.312780 5506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:10:31.314915 5693 heartbeater.cc:347] Connected to a master server at 127.3.69.62:34461
I20260806 09:10:31.315224 5693 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:31.315979 5693 heartbeater.cc:511] Master 127.3.69.62:34461 requested a full tablet report, sending...
I20260806 09:10:31.317343 5000 ts_manager.cc:194] Registered new tserver with Master: a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633)
I20260806 09:10:31.318202 5000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:57207
I20260806 09:10:31.319638 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 5506
I20260806 09:10:31.319841 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:10:31.320276 3348 external_mini_cluster.cc:1428] Setting key 6f495b40db17346c6bc5a43a5d6c54f8
I20260806 09:10:31.323304 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:10:31 dist-test-slave-w4v5 krb5kdc[4928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:10:31 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:31.375562 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.359843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.3.69.62:34461)
0806 09:10:31.360093 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.360101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.360324 (+ 223us) server_negotiation.cc:407] Connection header received
0806 09:10:31.360556 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.360561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.360650 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.360781 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:31.362297 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.363045 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.364449 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.364971 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.365110 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.369168 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:31.369173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:31.369177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:31.369213 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:31.372604 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:31.373345 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:31.373350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:31.373354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:31.373518 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:31.374093 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:31.374098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:31.374101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:31.374443 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:31.374635 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.374933 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.375192 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:10:31.379926 5000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:10:31.386911 5000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:31.410446 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.400422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.3.69.1:34121)
0806 09:10:31.402568 (+ 2146us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.402582 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.402617 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:10:31.402744 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.402755 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.403112 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.403318 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:31.404044 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.405603 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.407913 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.409329 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.409480 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.409722 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.409982 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.410145 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1937,"thread_start_us":1775,"threads_started":1}
I20260806 09:10:31.410607 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.399837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34121 (local address 127.0.0.1:53210)
0806 09:10:31.401066 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.401089 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.401217 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.403326 (+ 2109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.403332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.403363 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:31.403791 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.403809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.405755 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.405760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.407754 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.407766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.409526 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.409533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.409727 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.409794 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.409927 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.410123 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":71,"thread_start_us":927,"threads_started":1}
I20260806 09:10:31.411443 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.400325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:44639 (local address 127.0.0.1:35246)
0806 09:10:31.401842 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.401864 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.401980 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.403575 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.403579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.403599 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:31.404094 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.404109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.405858 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.405861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.408418 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.408433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.410469 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.410489 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.410705 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.410741 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.410838 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.410978 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"mutex_wait_us":103,"thread_start_us":1193,"threads_started":1}
I20260806 09:10:31.411549 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.399025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41633 (local address 127.0.0.1:37356)
0806 09:10:31.400019 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.400079 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.400295 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.403972 (+ 3677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.403976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.404004 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:31.404399 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.404409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.406407 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.406413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.408737 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.408750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.410597 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.410604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.410758 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.410795 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.410850 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.410978 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":35,"thread_start_us":421,"threads_started":1}
I20260806 09:10:31.412250 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.399278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.3.69.3:41633)
0806 09:10:31.403268 (+ 3990us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.403278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.403373 (+ 95us) server_negotiation.cc:407] Connection header received
0806 09:10:31.403471 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.403477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.403690 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.403958 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:31.404627 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.406250 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.408911 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.410343 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.411405 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.411641 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.411712 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.411887 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3823,"thread_start_us":3600,"threads_started":1}
I20260806 09:10:31.414456 5210 tablet_service.cc:1467] Processing CreateTablet for tablet df3bd59d1af54be781b5a30409b33fd3 (DEFAULT_TABLE table=test-table [id=89102c499e694b1bac2f51474f6c1038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:31.415309 5601 tablet_service.cc:1467] Processing CreateTablet for tablet df3bd59d1af54be781b5a30409b33fd3 (DEFAULT_TABLE table=test-table [id=89102c499e694b1bac2f51474f6c1038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:31.415802 5210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3bd59d1af54be781b5a30409b33fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.416531 5601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3bd59d1af54be781b5a30409b33fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.419726 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.400831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.3.69.2:44639)
0806 09:10:31.402758 (+ 1927us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.402769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.402803 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:10:31.402912 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.402920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.403248 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.403446 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:31.404303 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.405692 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.408597 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.409981 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.410106 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.410298 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.419212 (+ 8914us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.419427 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1772,"thread_start_us":1723,"threads_started":1}
I20260806 09:10:31.423243 5392 tablet_service.cc:1467] Processing CreateTablet for tablet df3bd59d1af54be781b5a30409b33fd3 (DEFAULT_TABLE table=test-table [id=89102c499e694b1bac2f51474f6c1038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:31.424649 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df3bd59d1af54be781b5a30409b33fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.428336 5728 tablet_bootstrap.cc:492] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Bootstrap starting.
I20260806 09:10:31.431844 5728 tablet_bootstrap.cc:654] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:31.433053 5728 log.cc:824] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:31.435585 5729 tablet_bootstrap.cc:492] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Bootstrap starting.
I20260806 09:10:31.437671 5728 tablet_bootstrap.cc:492] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: No bootstrap required, opened a new log
I20260806 09:10:31.438001 5728 ts_tablet_manager.cc:1397] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 09:10:31.439262 5729 tablet_bootstrap.cc:654] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:31.440639 5729 log.cc:824] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:31.441226 5731 tablet_bootstrap.cc:492] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Bootstrap starting.
I20260806 09:10:31.444229 5729 tablet_bootstrap.cc:492] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: No bootstrap required, opened a new log
I20260806 09:10:31.444491 5729 ts_tablet_manager.cc:1397] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260806 09:10:31.444656 5731 tablet_bootstrap.cc:654] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:31.445966 5731 log.cc:824] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:31.448518 5728 raft_consensus.cc:348] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.449076 5731 tablet_bootstrap.cc:492] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: No bootstrap required, opened a new log
I20260806 09:10:31.449067 5728 raft_consensus.cc:374] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:31.449178 5728 raft_consensus.cc:729] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec666f619064437b4a2949984067f1d, State: Initialized, Role: FOLLOWER
I20260806 09:10:31.449419 5731 ts_tablet_manager.cc:1397] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 09:10:31.449847 5728 consensus_queue.cc:260] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.452886 5729 raft_consensus.cc:348] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.453254 5729 raft_consensus.cc:374] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:31.453373 5729 raft_consensus.cc:729] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e126dcd72b426991ff8f6cf2193794, State: Initialized, Role: FOLLOWER
I20260806 09:10:31.454053 5729 consensus_queue.cc:260] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.454618 5728 ts_tablet_manager.cc:1428] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260806 09:10:31.456193 5693 heartbeater.cc:503] Master 127.3.69.62:34461 was elected leader, sending a full tablet report...
I20260806 09:10:31.456691 5729 ts_tablet_manager.cc:1428] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260806 09:10:31.462143 5731 raft_consensus.cc:348] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.462502 5731 raft_consensus.cc:374] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:31.462580 5731 raft_consensus.cc:729] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3774798ac00a4a25a537dcbee22aed8c, State: Initialized, Role: FOLLOWER
I20260806 09:10:31.463045 5731 consensus_queue.cc:260] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.464218 5500 heartbeater.cc:503] Master 127.3.69.62:34461 was elected leader, sending a full tablet report...
I20260806 09:10:31.464921 5731 ts_tablet_manager.cc:1428] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260806 09:10:31.499812 5306 tablet.cc:2369] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:10:31.541004 5501 tablet.cc:2369] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:10:31.564488 5694 tablet.cc:2369] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:10:31.566381 5737 raft_consensus.cc:482] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:31.566715 5737 raft_consensus.cc:504] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.568629 5737 leader_election.cc:288] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633), 3ec666f619064437b4a2949984067f1d (127.3.69.1:34121)
I20260806 09:10:31.576081 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.569116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:47167 (local address 127.3.69.1:34121)
0806 09:10:31.569376 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.569386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.570431 (+ 1045us) server_negotiation.cc:407] Connection header received
0806 09:10:31.570683 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.570687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.570760 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.570863 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:31.571657 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.572403 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.574255 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.575033 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.575251 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.575365 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.575693 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.575820 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:10:31.576339 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.569093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:36341 (local address 127.3.69.3:41633)
0806 09:10:31.569303 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.569311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.570488 (+ 1177us) server_negotiation.cc:407] Connection header received
0806 09:10:31.570734 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.570740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.570812 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.570911 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:31.571739 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.572522 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.574304 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.575311 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.575475 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.575590 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.575936 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.576074 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 09:10:31.576354 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.568960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34121 (local address 127.3.69.2:47167)
0806 09:10:31.570315 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.570341 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.570532 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.570878 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.570883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.570906 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:31.571494 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.571508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.572542 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.572547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.574095 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.574111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.575312 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.575318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.575527 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.575556 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.575668 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.575802 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":28,"thread_start_us":731,"threads_started":1}
I20260806 09:10:31.576740 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.568969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41633 (local address 127.3.69.2:36341)
0806 09:10:31.570363 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.570378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.570584 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.570924 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.570929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.570951 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:31.571574 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.571584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.572674 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.572679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.574142 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.574154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.575569 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.575576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.575731 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.575795 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.575908 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.576034 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"mutex_wait_us":191,"thread_start_us":715,"threads_started":1}
I20260806 09:10:31.577620 5244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3bd59d1af54be781b5a30409b33fd3" candidate_uuid: "3774798ac00a4a25a537dcbee22aed8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec666f619064437b4a2949984067f1d" is_pre_election: true
I20260806 09:10:31.577811 5629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3bd59d1af54be781b5a30409b33fd3" candidate_uuid: "3774798ac00a4a25a537dcbee22aed8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e126dcd72b426991ff8f6cf2193794" is_pre_election: true
I20260806 09:10:31.578300 5244 raft_consensus.cc:2454] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3774798ac00a4a25a537dcbee22aed8c in term 0.
I20260806 09:10:31.578419 5629 raft_consensus.cc:2454] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3774798ac00a4a25a537dcbee22aed8c in term 0.
I20260806 09:10:31.579171 5332 leader_election.cc:302] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3774798ac00a4a25a537dcbee22aed8c, 3ec666f619064437b4a2949984067f1d, a3e126dcd72b426991ff8f6cf2193794; no voters:
I20260806 09:10:31.579625 5737 raft_consensus.cc:2781] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:31.579792 5737 raft_consensus.cc:482] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:31.579928 5737 raft_consensus.cc:3037] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:31.582425 5737 raft_consensus.cc:504] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.583108 5737 leader_election.cc:288] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [CANDIDATE]: Term 1 election: Requested vote from peers a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633), 3ec666f619064437b4a2949984067f1d (127.3.69.1:34121)
I20260806 09:10:31.583503 5629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3bd59d1af54be781b5a30409b33fd3" candidate_uuid: "3774798ac00a4a25a537dcbee22aed8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e126dcd72b426991ff8f6cf2193794"
I20260806 09:10:31.583629 5244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df3bd59d1af54be781b5a30409b33fd3" candidate_uuid: "3774798ac00a4a25a537dcbee22aed8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec666f619064437b4a2949984067f1d"
I20260806 09:10:31.583724 5629 raft_consensus.cc:3037] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:31.583844 5244 raft_consensus.cc:3037] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:31.587127 5244 raft_consensus.cc:2454] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3774798ac00a4a25a537dcbee22aed8c in term 1.
I20260806 09:10:31.587162 5629 raft_consensus.cc:2454] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3774798ac00a4a25a537dcbee22aed8c in term 1.
I20260806 09:10:31.587549 5332 leader_election.cc:302] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3774798ac00a4a25a537dcbee22aed8c, 3ec666f619064437b4a2949984067f1d; no voters:
I20260806 09:10:31.587859 5737 raft_consensus.cc:2781] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:31.588419 5737 raft_consensus.cc:686] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 1 LEADER]: Becoming Leader. State: Replica: 3774798ac00a4a25a537dcbee22aed8c, State: Running, Role: LEADER
I20260806 09:10:31.588963 5737 consensus_queue.cc:237] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } }
I20260806 09:10:31.593619 5000 catalog_manager.cc:5495] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3774798ac00a4a25a537dcbee22aed8c (127.3.69.2). New cstate: current_term: 1 leader_uuid: "3774798ac00a4a25a537dcbee22aed8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } health_report { overall_health: UNKNOWN } } }
I20260806 09:10:31.660928 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.654880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.3.69.2:44639)
0806 09:10:31.655159 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.655167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.655235 (+ 68us) server_negotiation.cc:407] Connection header received
0806 09:10:31.655583 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.655588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.655682 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.655817 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:31.656163 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.656966 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.658440 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.658840 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.658943 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.659617 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:31.660243 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:31.660348 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.660544 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.660668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:10:31.670733 5244 raft_consensus.cc:1261] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Refusing update from remote peer 3774798ac00a4a25a537dcbee22aed8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:31.671802 5741 consensus_queue.cc:1035] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:31.673419 5629 raft_consensus.cc:1261] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Refusing update from remote peer 3774798ac00a4a25a537dcbee22aed8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:31.674340 5741 consensus_queue.cc:1035] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:31.683269 5745 mvcc.cc:205] Tried to move back new op lower bound from 7315486440110010368 to 7315486439792099328. Current Snapshot: MvccSnapshot[applied={T|T < 7315486440110010368}]
I20260806 09:10:31.684218 5746 mvcc.cc:205] Tried to move back new op lower bound from 7315486440110010368 to 7315486439792099328. Current Snapshot: MvccSnapshot[applied={T|T < 7315486440110010368}]
I20260806 09:10:31.689333 5749 mvcc.cc:205] Tried to move back new op lower bound from 7315486440110010368 to 7315486439792099328. Current Snapshot: MvccSnapshot[applied={T|T < 7315486440110010368}]
I20260806 09:10:31.705297 4998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:31.705665 4998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:31.709893 4998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8: Sending DeleteTablet for 3 replicas of tablet df3bd59d1af54be781b5a30409b33fd3
I20260806 09:10:31.711351 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet df3bd59d1af54be781b5a30409b33fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:31 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:35246
I20260806 09:10:31.711683 5210 tablet_service.cc:1514] Processing DeleteTablet for tablet df3bd59d1af54be781b5a30409b33fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:31 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:53210
I20260806 09:10:31.712898 4998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 09:10:31.713253 5601 tablet_service.cc:1514] Processing DeleteTablet for tablet df3bd59d1af54be781b5a30409b33fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:31 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37356
W20260806 09:10:31.713667 4998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:31.716616 5760 tablet_replica.cc:330] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: stopping tablet replica
I20260806 09:10:31.717226 5760 raft_consensus.cc:2229] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:31.717856 5760 raft_consensus.cc:2258] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:31.718681 5758 tablet_replica.cc:330] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: stopping tablet replica
I20260806 09:10:31.719120 5758 raft_consensus.cc:2229] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:10:31.720017 5758 raft_consensus.cc:2258] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:31.720247 5759 tablet_replica.cc:330] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: stopping tablet replica
I20260806 09:10:31.720669 5759 raft_consensus.cc:2229] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:31.722146 5760 ts_tablet_manager.cc:1905] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:31.722347 5759 raft_consensus.cc:2258] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:31.724705 5759 ts_tablet_manager.cc:1905] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:31.724927 5601 tablet_service.cc:1467] Processing CreateTablet for tablet 49d133e057434cd58d10bc407e053ade (DEFAULT_TABLE table=test-table [id=d3becb50911a4b12b8b024eecf2ef4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:31.725409 5601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d133e057434cd58d10bc407e053ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.729699 5760 ts_tablet_manager.cc:1918] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:10:31.729931 5760 log.cc:1196] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/df3bd59d1af54be781b5a30409b33fd3
I20260806 09:10:31.730648 5760 ts_tablet_manager.cc:1939] T df3bd59d1af54be781b5a30409b33fd3 P a3e126dcd72b426991ff8f6cf2193794: Deleting consensus metadata
I20260806 09:10:31.731675 5392 tablet_service.cc:1467] Processing CreateTablet for tablet 49d133e057434cd58d10bc407e053ade (DEFAULT_TABLE table=test-table [id=d3becb50911a4b12b8b024eecf2ef4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:31.732193 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d133e057434cd58d10bc407e053ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.732497 5759 ts_tablet_manager.cc:1918] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:10:31.732692 5759 log.cc:1196] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/df3bd59d1af54be781b5a30409b33fd3
I20260806 09:10:31.732712 4975 catalog_manager.cc:4841] TS a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633): tablet df3bd59d1af54be781b5a30409b33fd3 (table test-table [id=89102c499e694b1bac2f51474f6c1038]) successfully deleted
I20260806 09:10:31.733359 5759 ts_tablet_manager.cc:1939] T df3bd59d1af54be781b5a30409b33fd3 P 3ec666f619064437b4a2949984067f1d: Deleting consensus metadata
I20260806 09:10:31.735162 4976 catalog_manager.cc:4841] TS 3ec666f619064437b4a2949984067f1d (127.3.69.1:34121): tablet df3bd59d1af54be781b5a30409b33fd3 (table test-table [id=89102c499e694b1bac2f51474f6c1038]) successfully deleted
I20260806 09:10:31.735288 5210 tablet_service.cc:1467] Processing CreateTablet for tablet 49d133e057434cd58d10bc407e053ade (DEFAULT_TABLE table=test-table [id=d3becb50911a4b12b8b024eecf2ef4b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:31.735839 5210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d133e057434cd58d10bc407e053ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.736440 5729 tablet_bootstrap.cc:492] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: Bootstrap starting.
I20260806 09:10:31.738009 5758 ts_tablet_manager.cc:1905] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:31.738077 5729 tablet_bootstrap.cc:654] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:31.741348 5731 tablet_bootstrap.cc:492] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: Bootstrap starting.
I20260806 09:10:31.743317 5731 tablet_bootstrap.cc:654] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:31.744414 5729 tablet_bootstrap.cc:492] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: No bootstrap required, opened a new log
I20260806 09:10:31.744626 5729 ts_tablet_manager.cc:1397] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260806 09:10:31.744908 5728 tablet_bootstrap.cc:492] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: Bootstrap starting.
I20260806 09:10:31.745646 5729 raft_consensus.cc:348] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.745980 5729 raft_consensus.cc:374] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:31.746119 5729 raft_consensus.cc:729] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e126dcd72b426991ff8f6cf2193794, State: Initialized, Role: FOLLOWER
I20260806 09:10:31.746475 5729 consensus_queue.cc:260] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.746704 5728 tablet_bootstrap.cc:654] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:31.746768 5758 ts_tablet_manager.cc:1918] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:10:31.746948 5758 log.cc:1196] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/df3bd59d1af54be781b5a30409b33fd3
I20260806 09:10:31.747579 5758 ts_tablet_manager.cc:1939] T df3bd59d1af54be781b5a30409b33fd3 P 3774798ac00a4a25a537dcbee22aed8c: Deleting consensus metadata
I20260806 09:10:31.748235 5729 ts_tablet_manager.cc:1428] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260806 09:10:31.750831 4975 catalog_manager.cc:4841] TS 3774798ac00a4a25a537dcbee22aed8c (127.3.69.2:44639): tablet df3bd59d1af54be781b5a30409b33fd3 (table test-table [id=89102c499e694b1bac2f51474f6c1038]) successfully deleted
I20260806 09:10:31.750959 5728 tablet_bootstrap.cc:492] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: No bootstrap required, opened a new log
I20260806 09:10:31.751224 5728 ts_tablet_manager.cc:1397] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 09:10:31.751829 5731 tablet_bootstrap.cc:492] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: No bootstrap required, opened a new log
I20260806 09:10:31.752058 5731 ts_tablet_manager.cc:1397] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20260806 09:10:31.752099 5728 raft_consensus.cc:348] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.752338 5728 raft_consensus.cc:374] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:31.752441 5728 raft_consensus.cc:729] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec666f619064437b4a2949984067f1d, State: Initialized, Role: FOLLOWER
I20260806 09:10:31.752766 5728 consensus_queue.cc:260] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.752933 5731 raft_consensus.cc:348] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.753193 5731 raft_consensus.cc:374] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:31.753298 5731 raft_consensus.cc:729] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3774798ac00a4a25a537dcbee22aed8c, State: Initialized, Role: FOLLOWER
I20260806 09:10:31.753614 5731 consensus_queue.cc:260] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.754837 5731 ts_tablet_manager.cc:1428] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260806 09:10:31.755348 5728 ts_tablet_manager.cc:1428] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 09:10:31.867650 5735 raft_consensus.cc:482] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:31.867913 5735 raft_consensus.cc:504] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.869736 5735 leader_election.cc:288] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3774798ac00a4a25a537dcbee22aed8c (127.3.69.2:44639), a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633)
I20260806 09:10:31.877774 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.870398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:60921 (local address 127.3.69.2:44639)
0806 09:10:31.870643 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.870651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.870858 (+ 207us) server_negotiation.cc:407] Connection header received
0806 09:10:31.871108 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.871113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.871195 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.871296 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:31.872185 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.873431 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.875516 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.876688 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.876832 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.876932 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.877313 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.877458 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:10:31.877794 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.870131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:44639 (local address 127.3.69.1:60921)
0806 09:10:31.870752 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.870767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.870943 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.871482 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.871488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.871521 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:31.871967 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.871980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.873584 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.873590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.875325 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.875337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.876921 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.876928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.877108 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.877149 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.877261 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.877396 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1,"thread_start_us":308,"threads_started":1}
I20260806 09:10:31.878844 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.871336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:38035 (local address 127.3.69.3:41633)
0806 09:10:31.871523 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.871529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.872154 (+ 625us) server_negotiation.cc:407] Connection header received
0806 09:10:31.872397 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.872401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.872459 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.872556 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:31.873065 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.873955 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.876628 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.877686 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.877823 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.877948 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.878463 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.878592 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:10:31.878868 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.870946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41633 (local address 127.3.69.1:38035)
0806 09:10:31.872050 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.872068 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.872251 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.872570 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.872574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.872595 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:31.872922 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.872930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.874101 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.874110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.876407 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.876424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.878099 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.878106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.878302 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.878326 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.878430 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.878543 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":753,"threads_started":1}
I20260806 09:10:31.878962 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d133e057434cd58d10bc407e053ade" candidate_uuid: "3ec666f619064437b4a2949984067f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3774798ac00a4a25a537dcbee22aed8c" is_pre_election: true
I20260806 09:10:31.879494 5423 raft_consensus.cc:2454] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec666f619064437b4a2949984067f1d in term 0.
I20260806 09:10:31.880204 5126 leader_election.cc:302] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3774798ac00a4a25a537dcbee22aed8c, 3ec666f619064437b4a2949984067f1d; no voters:
I20260806 09:10:31.880666 5735 raft_consensus.cc:2781] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:31.880787 5735 raft_consensus.cc:482] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:31.880885 5735 raft_consensus.cc:3037] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:31.883553 5629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d133e057434cd58d10bc407e053ade" candidate_uuid: "3ec666f619064437b4a2949984067f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e126dcd72b426991ff8f6cf2193794" is_pre_election: true
I20260806 09:10:31.883816 5629 raft_consensus.cc:2454] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec666f619064437b4a2949984067f1d in term 0.
I20260806 09:10:31.884197 5735 raft_consensus.cc:504] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.885003 5735 leader_election.cc:288] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [CANDIDATE]: Term 1 election: Requested vote from peers 3774798ac00a4a25a537dcbee22aed8c (127.3.69.2:44639), a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633)
I20260806 09:10:31.885408 5629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d133e057434cd58d10bc407e053ade" candidate_uuid: "3ec666f619064437b4a2949984067f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e126dcd72b426991ff8f6cf2193794"
I20260806 09:10:31.885417 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d133e057434cd58d10bc407e053ade" candidate_uuid: "3ec666f619064437b4a2949984067f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3774798ac00a4a25a537dcbee22aed8c"
I20260806 09:10:31.885619 5629 raft_consensus.cc:3037] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:31.885630 5423 raft_consensus.cc:3037] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:31.888716 5629 raft_consensus.cc:2454] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec666f619064437b4a2949984067f1d in term 1.
I20260806 09:10:31.889086 5423 raft_consensus.cc:2454] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec666f619064437b4a2949984067f1d in term 1.
I20260806 09:10:31.889114 5126 leader_election.cc:302] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec666f619064437b4a2949984067f1d, a3e126dcd72b426991ff8f6cf2193794; no voters:
I20260806 09:10:31.889425 5735 raft_consensus.cc:2781] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:31.890028 5735 raft_consensus.cc:686] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 1 LEADER]: Becoming Leader. State: Replica: 3ec666f619064437b4a2949984067f1d, State: Running, Role: LEADER
I20260806 09:10:31.890443 5735 consensus_queue.cc:237] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:31.894258 4998 catalog_manager.cc:5495] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ec666f619064437b4a2949984067f1d (127.3.69.1). New cstate: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } health_report { overall_health: UNKNOWN } } }
Aug 06 09:10:31 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007429, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:31.975481 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.957874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34461 (local address 127.0.0.1:53820)
0806 09:10:31.958929 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:31.958951 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:31.959153 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:31.959520 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:31.959524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:31.960450 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:31.960725 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.960740 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.962021 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.962026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.963639 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:31.963654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.964496 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.964503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:31.964720 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.965590 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:31.965614 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:31.968866 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:31.972310 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:31.972323 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:31.972331 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:31.972928 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:31.973423 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:31.973428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:31.973432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:31.973608 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:31.974255 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:31.974264 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:31.974481 (+ 217us) client_negotiation.cc:769] Sending connection context
0806 09:10:31.974670 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 09:10:31.975022 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":719,"threads_started":1}
I20260806 09:10:31.975685 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:31.957912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.3.69.62:34461)
0806 09:10:31.958717 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:31.958727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:31.959052 (+ 325us) server_negotiation.cc:407] Connection header received
0806 09:10:31.959289 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:31.959294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:31.959354 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:31.959450 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:31.960943 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.961873 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.963788 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:31.964309 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:31.964407 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:31.969099 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:31.969105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:31.969108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:31.969139 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:31.972163 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:31.973096 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:31.973101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:31.973103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:31.973300 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:31.973768 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:31.973771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:31.973774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:31.974116 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:31.974236 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:31.974809 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 09:10:31.975241 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":613,"thread_start_us":378,"threads_started":1}
I20260806 09:10:31.981532 4998 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:53820:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:10:31.983134 4998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:31.990523 5210 tablet_service.cc:1467] Processing CreateTablet for tablet a3eff7a95c2841ec844948ba27bafdc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2abfc2f6a06c40dfaa370fb68685ca17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:10:31.991159 5210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3eff7a95c2841ec844948ba27bafdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.991343 5392 tablet_service.cc:1467] Processing CreateTablet for tablet a3eff7a95c2841ec844948ba27bafdc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2abfc2f6a06c40dfaa370fb68685ca17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:10:31.991614 5601 tablet_service.cc:1467] Processing CreateTablet for tablet a3eff7a95c2841ec844948ba27bafdc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2abfc2f6a06c40dfaa370fb68685ca17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:10:31.991942 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3eff7a95c2841ec844948ba27bafdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.992182 5601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3eff7a95c2841ec844948ba27bafdc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:31.998652 5728 tablet_bootstrap.cc:492] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: Bootstrap starting.
I20260806 09:10:31.999552 5731 tablet_bootstrap.cc:492] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: Bootstrap starting.
I20260806 09:10:31.999965 5729 tablet_bootstrap.cc:492] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: Bootstrap starting.
I20260806 09:10:32.000519 5728 tablet_bootstrap.cc:654] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:32.001389 5731 tablet_bootstrap.cc:654] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:32.001842 5729 tablet_bootstrap.cc:654] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:32.012049 5728 tablet_bootstrap.cc:492] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: No bootstrap required, opened a new log
I20260806 09:10:32.012254 5728 ts_tablet_manager.cc:1397] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260806 09:10:32.013247 5729 tablet_bootstrap.cc:492] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: No bootstrap required, opened a new log
I20260806 09:10:32.013141 5728 raft_consensus.cc:348] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.013411 5728 raft_consensus.cc:374] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:32.013494 5728 raft_consensus.cc:729] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec666f619064437b4a2949984067f1d, State: Initialized, Role: FOLLOWER
I20260806 09:10:32.013811 5731 tablet_bootstrap.cc:492] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: No bootstrap required, opened a new log
I20260806 09:10:32.014017 5731 ts_tablet_manager.cc:1397] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260806 09:10:32.013834 5728 consensus_queue.cc:260] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.014320 5735 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.014652 5735 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:32.014917 5729 ts_tablet_manager.cc:1397] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260806 09:10:32.015208 5728 ts_tablet_manager.cc:1428] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:10:32.015991 5729 raft_consensus.cc:348] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.016214 5729 raft_consensus.cc:374] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:32.016342 5729 raft_consensus.cc:729] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e126dcd72b426991ff8f6cf2193794, State: Initialized, Role: FOLLOWER
I20260806 09:10:32.016599 5729 consensus_queue.cc:260] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.017030 5736 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.017318 5736 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:32.017984 5731 raft_consensus.cc:348] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.018252 5731 raft_consensus.cc:374] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:32.018342 5731 raft_consensus.cc:729] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3774798ac00a4a25a537dcbee22aed8c, State: Initialized, Role: FOLLOWER
I20260806 09:10:32.018625 5731 consensus_queue.cc:260] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.020599 5729 ts_tablet_manager.cc:1428] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260806 09:10:32.020951 5731 ts_tablet_manager.cc:1428] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 09:10:32.020030 5737 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.021175 5737 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:32.209638 5735 raft_consensus.cc:482] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:32.209873 5735 raft_consensus.cc:504] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.210486 5735 leader_election.cc:288] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3774798ac00a4a25a537dcbee22aed8c (127.3.69.2:44639), a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633)
I20260806 09:10:32.211274 5629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3eff7a95c2841ec844948ba27bafdc7" candidate_uuid: "3ec666f619064437b4a2949984067f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e126dcd72b426991ff8f6cf2193794" is_pre_election: true
I20260806 09:10:32.211273 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3eff7a95c2841ec844948ba27bafdc7" candidate_uuid: "3ec666f619064437b4a2949984067f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3774798ac00a4a25a537dcbee22aed8c" is_pre_election: true
I20260806 09:10:32.211560 5629 raft_consensus.cc:2454] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec666f619064437b4a2949984067f1d in term 0.
I20260806 09:10:32.211561 5423 raft_consensus.cc:2454] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec666f619064437b4a2949984067f1d in term 0.
I20260806 09:10:32.212037 5126 leader_election.cc:302] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec666f619064437b4a2949984067f1d, a3e126dcd72b426991ff8f6cf2193794; no voters:
I20260806 09:10:32.212337 5735 raft_consensus.cc:2781] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:32.212445 5735 raft_consensus.cc:482] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:32.212553 5735 raft_consensus.cc:3037] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:32.214577 5735 raft_consensus.cc:504] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.215197 5735 leader_election.cc:288] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [CANDIDATE]: Term 1 election: Requested vote from peers 3774798ac00a4a25a537dcbee22aed8c (127.3.69.2:44639), a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633)
I20260806 09:10:32.215744 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3eff7a95c2841ec844948ba27bafdc7" candidate_uuid: "3ec666f619064437b4a2949984067f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3774798ac00a4a25a537dcbee22aed8c"
I20260806 09:10:32.215728 5629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3eff7a95c2841ec844948ba27bafdc7" candidate_uuid: "3ec666f619064437b4a2949984067f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e126dcd72b426991ff8f6cf2193794"
I20260806 09:10:32.215970 5629 raft_consensus.cc:3037] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:32.215970 5423 raft_consensus.cc:3037] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:32.218918 5423 raft_consensus.cc:2454] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec666f619064437b4a2949984067f1d in term 1.
I20260806 09:10:32.218918 5629 raft_consensus.cc:2454] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec666f619064437b4a2949984067f1d in term 1.
I20260806 09:10:32.219425 5126 leader_election.cc:302] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3774798ac00a4a25a537dcbee22aed8c, 3ec666f619064437b4a2949984067f1d; no voters:
I20260806 09:10:32.219691 5735 raft_consensus.cc:2781] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:32.219873 5735 raft_consensus.cc:686] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [term 1 LEADER]: Becoming Leader. State: Replica: 3ec666f619064437b4a2949984067f1d, State: Running, Role: LEADER
I20260806 09:10:32.220129 5735 consensus_queue.cc:237] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } }
I20260806 09:10:32.221873 5769 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: TxnStatusTablet state changed. Reason: New leader 3ec666f619064437b4a2949984067f1d. Latest consensus state: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.222167 5769 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:10:32.222960 5790 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:10:32.223291 5790 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:10:32.223716 5000 catalog_manager.cc:5495] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ec666f619064437b4a2949984067f1d (127.3.69.1). New cstate: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } health_report { overall_health: UNKNOWN } } }
I20260806 09:10:32.258229 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.251983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34121 (local address 127.0.0.1:53216)
0806 09:10:32.252240 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:32.252267 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:32.252418 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:32.252973 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:32.252980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:32.252996 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:10:32.253224 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:32.253233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.254523 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.254529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:32.255600 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:32.255612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.256436 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.256442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:32.256628 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.256682 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:10:32.257641 (+ 959us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:10:32.257663 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 09:10:32.257744 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 09:10:32.257893 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 09:10:32.258294 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.252071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.3.69.1:34121)
0806 09:10:32.252596 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:32.252602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:32.252630 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:10:32.252707 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:32.252711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:32.252779 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:32.252940 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:32.253414 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.254391 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.255742 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.256287 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.256426 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.256910 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:32.257529 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:32.257704 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:32.257785 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 09:10:32.257867 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":215,"threads_started":1}
I20260806 09:10:32.263882 5629 raft_consensus.cc:1261] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Refusing update from remote peer 3ec666f619064437b4a2949984067f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:32.264086 5423 raft_consensus.cc:1261] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Refusing update from remote peer 3ec666f619064437b4a2949984067f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:32.264798 5769 consensus_queue.cc:1035] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:32.265130 5735 consensus_queue.cc:1035] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:32.283994 5736 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: TxnStatusTablet state changed. Reason: New leader 3ec666f619064437b4a2949984067f1d. Latest consensus state: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.284376 5736 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:32.287261 5737 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: TxnStatusTablet state changed. Reason: New leader 3ec666f619064437b4a2949984067f1d. Latest consensus state: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.287593 5737 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:32.288558 5769 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.288903 5769 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:10:32.289191 5769 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.289565 5769 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P 3ec666f619064437b4a2949984067f1d: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:10:32.291652 5736 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.291965 5736 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P a3e126dcd72b426991ff8f6cf2193794: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:32.291994 5737 tablet_replica.cc:439] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ec666f619064437b4a2949984067f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec666f619064437b4a2949984067f1d" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34121 } } peers { permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 } } peers { permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 } } }
I20260806 09:10:32.294536 5737 tablet_replica.cc:442] T a3eff7a95c2841ec844948ba27bafdc7 P 3774798ac00a4a25a537dcbee22aed8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:32.309607 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.299198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.3.69.1:34121)
0806 09:10:32.299380 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:32.299386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:32.299405 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:10:32.299622 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:32.299628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:32.299721 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:32.299810 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:32.303060 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.304231 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.306865 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.307615 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.307770 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.308514 (+ 744us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:32.308678 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:32.308751 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:32.309030 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 09:10:32.309141 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:10:32.352689 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.332914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:51395 (local address 127.3.69.1:34121)
0806 09:10:32.333185 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:32.333191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:32.342740 (+ 9549us) server_negotiation.cc:407] Connection header received
0806 09:10:32.343479 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:32.343486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:32.343655 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:32.343758 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:32.345159 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.346963 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.350189 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.350967 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.351133 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.351789 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:32.351932 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:32.352028 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:32.352236 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 09:10:32.352320 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:10:32.352986 5816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.333326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34121 (local address 127.3.69.1:51395)
0806 09:10:32.342592 (+ 9266us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:32.342625 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:32.343224 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:32.344072 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:32.344080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:32.344093 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:10:32.344865 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:32.344881 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.347273 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.347279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:32.349990 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:32.350007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.351230 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.351239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:32.351485 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.351580 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:10:32.352065 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:10:32.352082 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 09:10:32.352173 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 09:10:32.352368 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8943,"thread_start_us":9426,"threads_started":1}
I20260806 09:10:32.366884 5423 raft_consensus.cc:1261] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Refusing update from remote peer 3ec666f619064437b4a2949984067f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:32.367715 5735 consensus_queue.cc:1035] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3774798ac00a4a25a537dcbee22aed8c" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:32.367992 5629 raft_consensus.cc:1261] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Refusing update from remote peer 3ec666f619064437b4a2949984067f1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:32.368821 5735 consensus_queue.cc:1035] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e126dcd72b426991ff8f6cf2193794" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:32.482824 4998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:32.483110 4998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:32.487012 4998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2ca53879962408a91c4a5b4794d85f8: Sending DeleteTablet for 3 replicas of tablet 49d133e057434cd58d10bc407e053ade
I20260806 09:10:32.487850 5210 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d133e057434cd58d10bc407e053ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:32 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:53210
I20260806 09:10:32.488736 5828 tablet_replica.cc:330] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: stopping tablet replica
I20260806 09:10:32.488942 5828 raft_consensus.cc:2229] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:10:32.489544 5828 raft_consensus.cc:2258] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:32.490784 5828 ts_tablet_manager.cc:1905] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:32.492331 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d133e057434cd58d10bc407e053ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:32 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:35246
I20260806 09:10:32.493037 5601 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d133e057434cd58d10bc407e053ade with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:32 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37356
I20260806 09:10:32.499784 5828 ts_tablet_manager.cc:1918] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:32.499967 5828 log.cc:1196] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/49d133e057434cd58d10bc407e053ade
I20260806 09:10:32.500566 5828 ts_tablet_manager.cc:1939] T 49d133e057434cd58d10bc407e053ade P 3ec666f619064437b4a2949984067f1d: Deleting consensus metadata
I20260806 09:10:32.502830 4976 catalog_manager.cc:4841] TS 3ec666f619064437b4a2949984067f1d (127.3.69.1:34121): tablet 49d133e057434cd58d10bc407e053ade (table test-table [id=d3becb50911a4b12b8b024eecf2ef4b8]) successfully deleted
Aug 06 09:10:32 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:32.532757 5830 tablet_replica.cc:330] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: stopping tablet replica
I20260806 09:10:32.533605 5829 tablet_replica.cc:330] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: stopping tablet replica
I20260806 09:10:32.533842 5829 raft_consensus.cc:2229] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:32.534104 5829 raft_consensus.cc:2258] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:32.532994 5830 raft_consensus.cc:2229] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:32.534539 5830 raft_consensus.cc:2258] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:32.535838 5830 ts_tablet_manager.cc:1905] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:32.540133 5829 ts_tablet_manager.cc:1905] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:32.544461 5830 ts_tablet_manager.cc:1918] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:32.544612 5830 log.cc:1196] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/49d133e057434cd58d10bc407e053ade
I20260806 09:10:32.545047 5830 ts_tablet_manager.cc:1939] T 49d133e057434cd58d10bc407e053ade P a3e126dcd72b426991ff8f6cf2193794: Deleting consensus metadata
I20260806 09:10:32.546136 4975 catalog_manager.cc:4841] TS a3e126dcd72b426991ff8f6cf2193794 (127.3.69.3:41633): tablet 49d133e057434cd58d10bc407e053ade (table test-table [id=d3becb50911a4b12b8b024eecf2ef4b8]) successfully deleted
I20260806 09:10:32.548681 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.509166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.3.69.1:34121)
0806 09:10:32.509420 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:32.509429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:32.517108 (+ 7679us) server_negotiation.cc:407] Connection header received
0806 09:10:32.517643 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:32.517652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:32.517745 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:32.517825 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:32.520623 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.521917 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.524642 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.525562 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.525682 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.532275 (+ 6593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:32.532296 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:32.532325 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:32.532386 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:32.536742 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.537656 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.537667 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.537672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.537879 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.538339 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.538344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.538348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.538867 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:32.547332 (+ 8465us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:32.548055 (+ 723us) server_negotiation.cc:299] Negotiation successful
0806 09:10:32.548306 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:10:32.548933 5829 ts_tablet_manager.cc:1918] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:32.549078 5829 log.cc:1196] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/49d133e057434cd58d10bc407e053ade
W20260806 09:10:32.550236 5296 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53244
I20260806 09:10:32.559403 5829 ts_tablet_manager.cc:1939] T 49d133e057434cd58d10bc407e053ade P 3774798ac00a4a25a537dcbee22aed8c: Deleting consensus metadata
I20260806 09:10:32.560765 4975 catalog_manager.cc:4841] TS 3774798ac00a4a25a537dcbee22aed8c (127.3.69.2:44639): tablet 49d133e057434cd58d10bc407e053ade (table test-table [id=d3becb50911a4b12b8b024eecf2ef4b8]) successfully deleted
I20260806 09:10:32.590314 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.554643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.3.69.62:34461)
0806 09:10:32.570881 (+ 16238us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:32.570890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:32.570914 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:10:32.570984 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:32.570990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:32.571189 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:32.571316 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:32.573547 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.575110 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.577315 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.578105 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.578241 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.582319 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:32.582329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:32.582335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:32.582400 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:32.586536 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.587583 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.587592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.587597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.587765 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.588265 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.588271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.588274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.588672 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:32.588924 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:32.589582 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 09:10:32.589843 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16069,"thread_start_us":16427,"threads_started":1}
W20260806 09:10:32.592162 4998 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53836
I20260806 09:10:32.594640 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 5096
I20260806 09:10:32.622099 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.595828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.3.69.1:34121)
0806 09:10:32.596035 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:32.596044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:32.603560 (+ 7516us) server_negotiation.cc:407] Connection header received
0806 09:10:32.603769 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:32.603776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:32.603867 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:32.603965 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:32.605766 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.607126 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.609188 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.610033 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.610162 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.613749 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:32.613759 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:32.613765 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:32.613811 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:32.618172 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.619110 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.619117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.619123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.619337 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.619858 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.619865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.619870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.620293 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:32.620603 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:32.621322 (+ 719us) server_negotiation.cc:299] Negotiation successful
0806 09:10:32.621592 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 09:10:32.623895 5296 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53260
W20260806 09:10:32.624526 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 5096: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:32.624712 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 5096
I20260806 09:10:32.665704 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 5310
Aug 06 09:10:32 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:10:32.682477 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.666523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.3.69.2:44639)
0806 09:10:32.667150 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:32.667156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:32.667175 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:10:32.667240 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:32.667243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:32.667302 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:32.667403 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:32.668908 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.669826 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.671724 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.672139 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.672244 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.677039 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:32.677053 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:32.677068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:32.677102 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:32.680066 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.680730 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.680737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.680739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.680864 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.681200 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.681203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.681205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.681583 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:32.681831 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:32.682049 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 09:10:32.682217 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":322,"threads_started":1}
W20260806 09:10:32.683302 5495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35270
W20260806 09:10:32.683996 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 5310: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:32.684159 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 5310
I20260806 09:10:32.717356 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 5506
Aug 06 09:10:32 dist-test-slave-w4v5 krb5kdc[4928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:10:32.732501 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:32.718184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.3.69.3:41633)
0806 09:10:32.718781 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:32.718786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:32.718804 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:10:32.718863 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:32.718866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:32.718929 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:32.719034 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:32.720313 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.721039 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.722251 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:32.722628 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:32.722722 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:32.726879 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:32.726893 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:32.726910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:32.726942 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:32.730004 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.730659 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.730666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.730668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.730799 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:32.731192 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:32.731197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:32.731199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:32.731543 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:32.731806 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:32.732012 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 09:10:32.732176 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":339,"threads_started":1}
W20260806 09:10:32.733279 5686 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37370
W20260806 09:10:32.733992 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 5506: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:32.734161 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 5506
I20260806 09:10:32.761924 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4949
I20260806 09:10:32.766464 5052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:32.883491 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4949
2026-08-06T09:10:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5183 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:32 dist-test-slave-w4v5 krb5kdc[5849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:32 dist-test-slave-w4v5 krb5kdc[5849](info): set up 2 sockets
Aug 06 09:10:32 dist-test-slave-w4v5 krb5kdc[5849](info): commencing operation
krb5kdc: starting...
W20260806 09:10:34.182956 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:34 dist-test-slave-w4v5 krb5kdc[5849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:34.391760 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37109
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:37109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:34.719806 5866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:34.720077 5866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:34.720275 5866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:34.727442 5866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:34.727543 5866 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:10:34.727614 5866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:34.727720 5866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:34.727775 5866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:34.727833 5866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:34.737336 5866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:37109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:34.738554 5866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:34.739904 5866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:34.751701 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:34.751567 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:34.751827 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:34.752943 5866 server_base.cc:1034] running on GCE node
I20260806 09:10:34.753751 5866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:34.755028 5866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:34.756306 5866 hybrid_clock.cc:648] HybridClock initialized: now 1786007434756254 us; error 32 us; skew 500 ppm
I20260806 09:10:34.756731 5866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:34 dist-test-slave-w4v5 krb5kdc[5849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007434, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:34.761297 5866 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:34.763962 5866 webserver.cc:460] Webserver started at http://127.3.69.62:42913/ using document root <none> and password file <none>
I20260806 09:10:34.764683 5866 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:34.764812 5866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:34.765148 5866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:34.768198 5866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "b21887c166da4205a4fc5b4e32a19ba8"
format_stamp: "Formatted at 2026-08-06 09:10:34 on dist-test-slave-w4v5"
server_key: "3eb2fad8efc27f0a422c895851e06e21"
server_key_iv: "7fd5ac9af59a93f42c12b7e72ac6e0d5"
server_key_version: "encryptionkey@0"
I20260806 09:10:34.768985 5866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "b21887c166da4205a4fc5b4e32a19ba8"
format_stamp: "Formatted at 2026-08-06 09:10:34 on dist-test-slave-w4v5"
server_key: "3eb2fad8efc27f0a422c895851e06e21"
server_key_iv: "7fd5ac9af59a93f42c12b7e72ac6e0d5"
server_key_version: "encryptionkey@0"
I20260806 09:10:34.774150 5866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:10:34.777633 5888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:34.778884 5866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:10:34.779122 5866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "b21887c166da4205a4fc5b4e32a19ba8"
format_stamp: "Formatted at 2026-08-06 09:10:34 on dist-test-slave-w4v5"
server_key: "3eb2fad8efc27f0a422c895851e06e21"
server_key_iv: "7fd5ac9af59a93f42c12b7e72ac6e0d5"
server_key_version: "encryptionkey@0"
I20260806 09:10:34.779345 5866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:34.794169 5866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:34.798415 5866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:34.798640 5866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:34.820757 5866 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:37109
I20260806 09:10:34.820776 5974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:37109 every 8 connection(s)
I20260806 09:10:34.822165 5866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:34.826050 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:34.830899 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 5866
I20260806 09:10:34.831156 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:34.831805 3348 external_mini_cluster.cc:1428] Setting key 1498d0f2c5e855206806a3727bca440b
I20260806 09:10:34.837153 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: Bootstrap starting.
I20260806 09:10:34.840932 5975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:34.842185 5975 log.cc:824] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:34.845492 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: No bootstrap required, opened a new log
Aug 06 09:10:34 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:34.856204 5975 raft_consensus.cc:348] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21887c166da4205a4fc5b4e32a19ba8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37109 } }
I20260806 09:10:34.856671 5975 raft_consensus.cc:374] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:34.856837 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:34.833895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.3.69.62:37109)
0806 09:10:34.834848 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:34.834861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:34.834905 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:10:34.836688 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:34.836711 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:34.837090 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:34.837456 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:34.839308 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:34.840656 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:34.842463 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:34.843312 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:34.843458 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:34.847820 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:34.847830 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:34.847849 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:34.847899 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:34.852124 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:34.852857 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:34.852871 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:34.852886 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:34.853118 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:34.853486 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:34.853491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:34.853495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:34.854271 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:34.854590 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:34.854969 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 09:10:34.855471 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":398,"threads_started":1}
I20260806 09:10:34.856798 5975 raft_consensus.cc:729] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b21887c166da4205a4fc5b4e32a19ba8, State: Initialized, Role: FOLLOWER
I20260806 09:10:34.857486 5975 consensus_queue.cc:260] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21887c166da4205a4fc5b4e32a19ba8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37109 } }
I20260806 09:10:34.857738 5975 raft_consensus.cc:388] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:34.857877 5975 raft_consensus.cc:482] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:34.858072 5975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:34.860942 5975 raft_consensus.cc:504] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21887c166da4205a4fc5b4e32a19ba8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37109 } }
I20260806 09:10:34.861457 5975 leader_election.cc:302] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b21887c166da4205a4fc5b4e32a19ba8; no voters:
I20260806 09:10:34.862129 5975 leader_election.cc:288] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:34.862311 5984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:34.863066 5984 raft_consensus.cc:686] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [term 1 LEADER]: Becoming Leader. State: Replica: b21887c166da4205a4fc5b4e32a19ba8, State: Running, Role: LEADER
I20260806 09:10:34.863611 5984 consensus_queue.cc:237] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21887c166da4205a4fc5b4e32a19ba8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37109 } }
I20260806 09:10:34.864272 5975 sys_catalog.cc:564] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:34.866654 5985 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b21887c166da4205a4fc5b4e32a19ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21887c166da4205a4fc5b4e32a19ba8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37109 } } }
I20260806 09:10:34.866559 5986 sys_catalog.cc:455] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b21887c166da4205a4fc5b4e32a19ba8. Latest consensus state: current_term: 1 leader_uuid: "b21887c166da4205a4fc5b4e32a19ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b21887c166da4205a4fc5b4e32a19ba8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37109 } } }
I20260806 09:10:34.867005 5985 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:34.867069 5986 sys_catalog.cc:458] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:34.868580 5995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:34.873831 5995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:34.883105 5995 catalog_manager.cc:1347] Generated new cluster ID: 35e396b59ec44437903f473f075c5dff
I20260806 09:10:34.883234 5995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:34.914072 5995 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:34.915607 5995 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:34.924638 5995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: Generated new TSK 0
I20260806 09:10:34.925472 5995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:10:34.981215 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37109
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:35.327888 6009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:35.328166 6009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:35.328343 6009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:35.335764 6009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:35.335873 6009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:35.336040 6009 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:10:35.336123 6009 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:10:35.336263 6009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:10:35.346025 6009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:37109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:35.347332 6009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:35.348693 6009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:35.362083 6018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:35.362140 6021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:35.362164 6017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:35.367218 6009 server_base.cc:1034] running on GCE node
I20260806 09:10:35.367928 6009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:35.368955 6009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:35.370172 6009 hybrid_clock.cc:648] HybridClock initialized: now 1786007435370103 us; error 59 us; skew 500 ppm
I20260806 09:10:35.370504 6009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:35 dist-test-slave-w4v5 krb5kdc[5849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007435, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:35.373759 6009 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:10:35.375366 6009 webserver.cc:460] Webserver started at http://127.3.69.1:39301/ using document root <none> and password file <none>
I20260806 09:10:35.376050 6009 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:35.376196 6009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:35.376515 6009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:35.379465 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "2bf15e6e8be34145b453ff076325278c"
format_stamp: "Formatted at 2026-08-06 09:10:35 on dist-test-slave-w4v5"
server_key: "fdcfea8d083163365727bb33ce22f66e"
server_key_iv: "48ebebdc140278034865ac1266039561"
server_key_version: "encryptionkey@0"
I20260806 09:10:35.380223 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "2bf15e6e8be34145b453ff076325278c"
format_stamp: "Formatted at 2026-08-06 09:10:35 on dist-test-slave-w4v5"
server_key: "fdcfea8d083163365727bb33ce22f66e"
server_key_iv: "48ebebdc140278034865ac1266039561"
server_key_version: "encryptionkey@0"
I20260806 09:10:35.385480 6009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 09:10:35.389838 6033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:35.391191 6009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:10:35.391356 6009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "2bf15e6e8be34145b453ff076325278c"
format_stamp: "Formatted at 2026-08-06 09:10:35 on dist-test-slave-w4v5"
server_key: "fdcfea8d083163365727bb33ce22f66e"
server_key_iv: "48ebebdc140278034865ac1266039561"
server_key_version: "encryptionkey@0"
I20260806 09:10:35.391554 6009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:35.414300 6009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:35.418565 6009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:35.418876 6009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:35.421567 6009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:35.421727 6009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:35.421870 6009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:35.421965 6009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:10:35 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007435, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:35.445319 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:35.422456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:56391 (local address 127.3.69.62:37109)
0806 09:10:35.422955 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:35.422962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:35.424376 (+ 1414us) server_negotiation.cc:407] Connection header received
0806 09:10:35.426556 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:35.426563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:35.426666 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:35.426795 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:35.429288 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.430652 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.432990 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.433414 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.433562 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:35.438616 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:35.438633 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:35.438639 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:35.438688 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:35.441746 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:35.442726 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:35.442732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:35.442736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:35.442913 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:35.443542 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:35.443547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:35.443551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:35.443829 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:35.443988 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:35.444703 (+ 715us) server_negotiation.cc:299] Negotiation successful
0806 09:10:35.444928 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":271,"threads_started":1}
I20260806 09:10:35.446434 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:35.423243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37109 (local address 127.3.69.1:56391)
0806 09:10:35.424198 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:35.424255 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:35.426268 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:35.427311 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:35.427326 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:35.428172 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:35.429003 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:35.429026 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.430878 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.430881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:35.432837 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:35.432849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.433729 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.433739 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:35.434105 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:35.434904 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:35.434938 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:35.438355 (+ 3417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:35.441892 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:35.441903 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:35.441927 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:35.442557 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:35.443105 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:35.443113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:35.443118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:35.443407 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:35.443962 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:35.443971 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:35.444326 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 09:10:35.444587 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 09:10:35.445107 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":412,"threads_started":1}
I20260806 09:10:35.466581 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:35.453796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:44417 (local address 127.3.69.62:37109)
0806 09:10:35.453957 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:35.453961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:35.454699 (+ 738us) server_negotiation.cc:407] Connection header received
0806 09:10:35.454913 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:35.454918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:35.455023 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:35.455154 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:35.456211 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.456895 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.458226 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.458636 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.458762 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:35.461057 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:35.461061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:35.461065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:35.461093 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:35.463924 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:35.464576 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:35.464584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:35.464587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:35.464719 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:35.465112 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:35.465118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:35.465122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:35.465399 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:35.465556 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:35.465987 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 09:10:35.466211 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 09:10:35.466674 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:35.453873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37109 (local address 127.3.69.1:44417)
0806 09:10:35.454564 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:35.454583 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:35.454777 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:35.455347 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:35.455352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:35.455776 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:35.456056 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:35.456064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.457038 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.457044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:35.458072 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:35.458084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.458833 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.458839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:35.458987 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:35.459703 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:35.459731 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:35.460859 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:35.464060 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:35.464066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:35.464070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:35.464460 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:35.464830 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:35.464835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:35.464838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:35.464999 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:35.465519 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:35.465524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:35.465743 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 09:10:35.465886 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 09:10:35.466206 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":440,"threads_started":1}
I20260806 09:10:35.484473 6009 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:34777
I20260806 09:10:35.484512 6207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:34777 every 8 connection(s)
I20260806 09:10:35.485812 6009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:10:35.486281 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 6009
I20260806 09:10:35.486539 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:10:35.487560 3348 external_mini_cluster.cc:1428] Setting key d7e5c0a7221b491c7d0d9119e408dc44
I20260806 09:10:35.488127 6208 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37109
I20260806 09:10:35.488397 6208 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:35.489063 6208 heartbeater.cc:511] Master 127.3.69.62:37109 requested a full tablet report, sending...
I20260806 09:10:35.491222 5916 ts_manager.cc:194] Registered new tserver with Master: 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777)
I20260806 09:10:35.492656 5916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:56391
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:10:35.531951 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37109
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:35.854993 6212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:35.855281 6212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:35.855476 6212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:35.862613 6212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:35.862749 6212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:35.862916 6212 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:10:35.862998 6212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:10:35.863155 6212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:10:35.872865 6212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:37109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:35.874099 6212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:35.875473 6212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:35.888628 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:35.888720 6218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:35.888682 6219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:35.889846 6212 server_base.cc:1034] running on GCE node
I20260806 09:10:35.890447 6212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:35.891462 6212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:35.892691 6212 hybrid_clock.cc:648] HybridClock initialized: now 1786007435892645 us; error 30 us; skew 500 ppm
I20260806 09:10:35.893038 6212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:35 dist-test-slave-w4v5 krb5kdc[5849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007435, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:35.900761 6212 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:10:35.902307 6212 webserver.cc:460] Webserver started at http://127.3.69.2:44013/ using document root <none> and password file <none>
I20260806 09:10:35.902968 6212 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:35.903120 6212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:35.903420 6212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:35.906286 6212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "1127b0b9f69145a39405c83ee9ea50e4"
format_stamp: "Formatted at 2026-08-06 09:10:35 on dist-test-slave-w4v5"
server_key: "4a790accaf3acfc7769a8a7e29fb961c"
server_key_iv: "1cd10ae6899ce0eac741a49a08408cc3"
server_key_version: "encryptionkey@0"
I20260806 09:10:35.907003 6212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "1127b0b9f69145a39405c83ee9ea50e4"
format_stamp: "Formatted at 2026-08-06 09:10:35 on dist-test-slave-w4v5"
server_key: "4a790accaf3acfc7769a8a7e29fb961c"
server_key_iv: "1cd10ae6899ce0eac741a49a08408cc3"
server_key_version: "encryptionkey@0"
I20260806 09:10:35.912261 6212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 09:10:35.915711 6230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:35.917011 6212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:10:35.917182 6212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "1127b0b9f69145a39405c83ee9ea50e4"
format_stamp: "Formatted at 2026-08-06 09:10:35 on dist-test-slave-w4v5"
server_key: "4a790accaf3acfc7769a8a7e29fb961c"
server_key_iv: "1cd10ae6899ce0eac741a49a08408cc3"
server_key_version: "encryptionkey@0"
I20260806 09:10:35.917387 6212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:35.945268 6212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:35.949746 6212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:35.950027 6212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:35.952711 6212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:35.952845 6212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:35.952996 6212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:35.953099 6212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:10:35 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007435, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:35.977447 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:35.953083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:57725 (local address 127.3.69.62:37109)
0806 09:10:35.953311 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:35.953320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:35.954907 (+ 1587us) server_negotiation.cc:407] Connection header received
0806 09:10:35.957943 (+ 3036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:35.957949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:35.958042 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:35.958163 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:35.960802 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.961731 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.964374 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.965099 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.965246 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:35.970016 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:35.970021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:35.970025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:35.970086 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:35.973088 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:35.974239 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:35.974247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:35.974253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:35.974418 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:35.975207 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:35.975213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:35.975216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:35.975668 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:35.975832 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:35.976864 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0806 09:10:35.977096 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:10:35.978401 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:35.953781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37109 (local address 127.3.69.2:57725)
0806 09:10:35.954717 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:35.954771 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:35.957584 (+ 2813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:35.958485 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:35.958498 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:35.959444 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:35.960557 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:35.960599 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.961961 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.961968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:35.964209 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:35.964232 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.965365 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.965374 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:35.965672 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:35.966354 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:35.966377 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:35.969730 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:35.973245 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:35.973258 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:35.973284 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:35.974054 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:35.974582 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:35.974587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:35.974592 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:35.974991 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:35.975901 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:35.975910 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:35.976413 (+ 503us) client_negotiation.cc:769] Sending connection context
0806 09:10:35.976754 (+ 341us) client_negotiation.cc:240] Negotiation successful
0806 09:10:35.977255 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":397,"threads_started":1}
I20260806 09:10:36.004959 6212 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:45565
I20260806 09:10:36.005046 6392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:45565 every 8 connection(s)
I20260806 09:10:36.006325 6212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:10:36.006887 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:35.991315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:49351 (local address 127.3.69.62:37109)
0806 09:10:35.991508 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:35.991514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:35.992155 (+ 641us) server_negotiation.cc:407] Connection header received
0806 09:10:35.992381 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:35.992389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:35.992478 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:35.992605 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:35.994028 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.995172 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.996811 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.997424 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.997664 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.000574 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:36.000582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:36.000589 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:36.000633 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:36.003809 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.004656 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.004663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.004667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.004848 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.005366 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.005371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.005375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.005723 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:36.005885 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.006289 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.006535 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:10:36.007012 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:35.991253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37109 (local address 127.3.69.2:49351)
0806 09:10:35.992020 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:35.992046 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:35.992221 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:35.992625 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:35.992634 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:35.993335 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:35.993859 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:35.993883 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.995313 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.995319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:35.996583 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:35.996599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:35.997985 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:35.997996 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:35.998231 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:35.999063 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:35.999132 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:36.000387 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:36.003947 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:36.003954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:36.003961 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:36.004516 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:36.005011 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:36.005017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:36.005022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:36.005244 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:36.005853 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:36.005857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:36.006055 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.006198 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.006541 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":389,"threads_started":1}
I20260806 09:10:36.009217 6393 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37109
I20260806 09:10:36.009461 6393 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:36.010274 6393 heartbeater.cc:511] Master 127.3.69.62:37109 requested a full tablet report, sending...
I20260806 09:10:36.010948 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 6212
I20260806 09:10:36.011157 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:10:36.011745 3348 external_mini_cluster.cc:1428] Setting key 605320e68510e5ed5cb0a05403d1bc36
I20260806 09:10:36.011974 5916 ts_manager.cc:194] Registered new tserver with Master: 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565)
I20260806 09:10:36.012992 5916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:57725
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:10:36.058399 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37109
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:32959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:36.386293 6398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:36.386552 6398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:36.386653 6398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:36.394855 6398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:36.394966 6398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:36.395114 6398 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:10:36.395210 6398 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:10:36.395285 6398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:10:36.404670 6398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:32959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:37109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:36.405836 6398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:36.407184 6398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:36.420686 6405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:36.420609 6404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:36.421026 6407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:36.421504 6398 server_base.cc:1034] running on GCE node
I20260806 09:10:36.422205 6398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:36.423250 6398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:36.424463 6398 hybrid_clock.cc:648] HybridClock initialized: now 1786007436424430 us; error 30 us; skew 500 ppm
I20260806 09:10:36.424805 6398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:36 dist-test-slave-w4v5 krb5kdc[5849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007436, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:36.428144 6398 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:10:36.429646 6398 webserver.cc:460] Webserver started at http://127.3.69.3:33887/ using document root <none> and password file <none>
I20260806 09:10:36.430357 6398 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:36.430522 6398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:36.430845 6398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:36.433905 6398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "f1a73751b43f4ae78cea34ef099dbe88"
format_stamp: "Formatted at 2026-08-06 09:10:36 on dist-test-slave-w4v5"
server_key: "c2e9a3a03ecd15b1b0211501fca68aba"
server_key_iv: "a57ceaa705610ff8fa03f21b46da61c0"
server_key_version: "encryptionkey@0"
I20260806 09:10:36.434731 6398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "f1a73751b43f4ae78cea34ef099dbe88"
format_stamp: "Formatted at 2026-08-06 09:10:36 on dist-test-slave-w4v5"
server_key: "c2e9a3a03ecd15b1b0211501fca68aba"
server_key_iv: "a57ceaa705610ff8fa03f21b46da61c0"
server_key_version: "encryptionkey@0"
I20260806 09:10:36.439976 6398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 09:10:36.443367 6418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:36.444592 6398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:10:36.444763 6398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "f1a73751b43f4ae78cea34ef099dbe88"
format_stamp: "Formatted at 2026-08-06 09:10:36 on dist-test-slave-w4v5"
server_key: "c2e9a3a03ecd15b1b0211501fca68aba"
server_key_iv: "a57ceaa705610ff8fa03f21b46da61c0"
server_key_version: "encryptionkey@0"
I20260806 09:10:36.444964 6398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:36.466805 6398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:36.470772 6398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:36.471063 6398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:36.473765 6398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:36.473894 6398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:36.474045 6398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:36.474162 6398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:10:36 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007436, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:36.495904 6208 heartbeater.cc:503] Master 127.3.69.62:37109 was elected leader, sending a full tablet report...
I20260806 09:10:36.496222 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.474140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:47067 (local address 127.3.69.62:37109)
0806 09:10:36.474369 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.474378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.475977 (+ 1599us) server_negotiation.cc:407] Connection header received
0806 09:10:36.477975 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.477984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.478082 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.478211 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:36.480549 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.481914 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.484609 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.485160 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.485296 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.488847 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:36.488852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:36.488855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:36.488886 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:36.492128 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.493126 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.493134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.493140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.493330 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.493943 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.493951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.493956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.494419 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:36.494598 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.495375 (+ 777us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.495701 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:10:36.497030 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.474815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37109 (local address 127.3.69.3:47067)
0806 09:10:36.475794 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:36.475847 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:36.477716 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:36.478523 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:36.478534 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:36.479477 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:36.480319 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.480342 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.482139 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.482150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.484457 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.484469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.485421 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.485429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.485633 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.486283 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:36.486309 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:36.488623 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:36.492334 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:36.492346 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:36.492368 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:36.492945 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:36.493466 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:36.493471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:36.493475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:36.493781 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:36.494582 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:36.494590 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:36.494894 (+ 304us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.495196 (+ 302us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.495647 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":485,"threads_started":1}
I20260806 09:10:36.518332 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.502531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:56439 (local address 127.3.69.62:37109)
0806 09:10:36.502748 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.502756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.503467 (+ 711us) server_negotiation.cc:407] Connection header received
0806 09:10:36.503647 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.503651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.503712 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.503788 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:36.505184 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.505873 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.508349 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.508844 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.508990 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.511545 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:36.511554 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:36.511562 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:36.511613 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:36.515026 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.516194 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.516202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.516207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.516365 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.516853 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.516860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.516864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.517169 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:36.517337 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.517761 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.517973 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:10:36.518422 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.502526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37109 (local address 127.3.69.3:56439)
0806 09:10:36.503344 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:36.503369 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:36.503531 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:36.503854 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:36.503862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:36.504564 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:36.505034 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.505050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.506008 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.506017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.508187 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.508208 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.509064 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.509074 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.509321 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.510101 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:36.510129 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:36.511337 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:36.515281 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:36.515286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:36.515290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:36.516025 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:36.516502 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:36.516507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:36.516511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:36.516669 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:36.517299 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:36.517303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:36.517490 (+ 187us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.517626 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.517930 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":399,"threads_started":1}
I20260806 09:10:36.535003 6398 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:45649
I20260806 09:10:36.535048 6591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:45649 every 8 connection(s)
I20260806 09:10:36.536311 6398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:10:36.538522 6592 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37109
I20260806 09:10:36.538725 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 6398
I20260806 09:10:36.538775 6592 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:36.538934 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:10:36.539561 3348 external_mini_cluster.cc:1428] Setting key e8c3898a14e73f9b9a0b3f2bd68ca090
I20260806 09:10:36.539572 6592 heartbeater.cc:511] Master 127.3.69.62:37109 requested a full tablet report, sending...
I20260806 09:10:36.541067 5916 ts_manager.cc:194] Registered new tserver with Master: f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649)
I20260806 09:10:36.541966 5916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:47067
I20260806 09:10:36.542774 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:10:36 dist-test-slave-w4v5 krb5kdc[5849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:10:36 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:36.597050 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.583293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.3.69.62:37109)
0806 09:10:36.583536 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.583543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.583731 (+ 188us) server_negotiation.cc:407] Connection header received
0806 09:10:36.583955 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.583968 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.584036 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.584149 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:36.585563 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.586305 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.587644 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.588081 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.588232 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.592150 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:36.592154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:36.592158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:36.592192 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:36.594664 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.595311 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.595315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.595317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.595491 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.595909 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.595912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.595914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.596176 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:36.596422 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.596613 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.596789 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:10:36.601296 5916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:10:36.608187 5916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:36.631664 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.619953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.3.69.2:45565)
0806 09:10:36.620504 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.620514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.621205 (+ 691us) server_negotiation.cc:407] Connection header received
0806 09:10:36.621724 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.621733 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.621952 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.622156 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:36.623222 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.625186 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.628981 (+ 3795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.630359 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.630519 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.630751 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.631145 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.631346 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
I20260806 09:10:36.632489 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.619989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34777 (local address 127.0.0.1:41096)
0806 09:10:36.621901 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:36.621921 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:36.622123 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:36.623184 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:36.623192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:36.623226 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:36.623781 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.623797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.626211 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.626220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.628732 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.628744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.631471 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.631478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.631640 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.631680 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.631794 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.631983 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1785,"mutex_wait_us":2,"thread_start_us":568,"threads_started":1}
I20260806 09:10:36.632730 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.620356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.3.69.1:34777)
0806 09:10:36.620892 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.620901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.622273 (+ 1372us) server_negotiation.cc:407] Connection header received
0806 09:10:36.622403 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.622413 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.622680 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.622897 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:36.624024 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.625991 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.628881 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.630356 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.631107 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.631313 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.632228 (+ 915us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.632396 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":847,"threads_started":1}
I20260806 09:10:36.632519 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.619769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:45565 (local address 127.0.0.1:45964)
0806 09:10:36.621051 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:36.621098 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:36.621351 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:36.622419 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:36.622425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:36.622456 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:36.622927 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.622945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.625435 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.625443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.628773 (+ 3330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.628790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.630592 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.630598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.630804 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.630879 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.631948 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.632085 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":492,"thread_start_us":1828,"threads_started":1}
I20260806 09:10:36.635843 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.622042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:45649 (local address 127.0.0.1:55276)
0806 09:10:36.627833 (+ 5791us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:36.627850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:36.628042 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:36.628935 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:36.628939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:36.628967 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:36.629237 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.629244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.630796 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.630801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.633003 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.633021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.635065 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.635091 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.635269 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.635299 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.635400 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.635517 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5671,"thread_start_us":5519,"threads_started":1}
I20260806 09:10:36.635916 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.622408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.3.69.3:45649)
0806 09:10:36.627393 (+ 4985us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.627402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.627946 (+ 544us) server_negotiation.cc:407] Connection header received
0806 09:10:36.628225 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.628232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.628439 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.628587 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:36.629440 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.630620 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.633197 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.634555 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.634656 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.634892 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.635452 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.635608 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4825,"thread_start_us":4637,"threads_started":1}
I20260806 09:10:36.636629 6298 tablet_service.cc:1467] Processing CreateTablet for tablet 612f1cc160ec4bf2b06266535d2c0c3c (DEFAULT_TABLE table=test-table [id=4bf587808d9a4659b107d6ea0ab031e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:36.636672 6115 tablet_service.cc:1467] Processing CreateTablet for tablet 612f1cc160ec4bf2b06266535d2c0c3c (DEFAULT_TABLE table=test-table [id=4bf587808d9a4659b107d6ea0ab031e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:36.638109 6115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612f1cc160ec4bf2b06266535d2c0c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:36.638113 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612f1cc160ec4bf2b06266535d2c0c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:36.638970 6497 tablet_service.cc:1467] Processing CreateTablet for tablet 612f1cc160ec4bf2b06266535d2c0c3c (DEFAULT_TABLE table=test-table [id=4bf587808d9a4659b107d6ea0ab031e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:36.640194 6497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 612f1cc160ec4bf2b06266535d2c0c3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:36.651743 6614 tablet_bootstrap.cc:492] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Bootstrap starting.
I20260806 09:10:36.653878 6616 tablet_bootstrap.cc:492] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Bootstrap starting.
I20260806 09:10:36.655750 6614 tablet_bootstrap.cc:654] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:36.655769 6618 tablet_bootstrap.cc:492] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Bootstrap starting.
I20260806 09:10:36.657249 6616 tablet_bootstrap.cc:654] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:36.657207 6614 log.cc:824] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:36.658588 6616 log.cc:824] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:36.658946 6618 tablet_bootstrap.cc:654] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:36.660238 6618 log.cc:824] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:36.661725 6616 tablet_bootstrap.cc:492] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: No bootstrap required, opened a new log
I20260806 09:10:36.661731 6614 tablet_bootstrap.cc:492] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: No bootstrap required, opened a new log
I20260806 09:10:36.661988 6616 ts_tablet_manager.cc:1397] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 09:10:36.662022 6614 ts_tablet_manager.cc:1397] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260806 09:10:36.663414 6618 tablet_bootstrap.cc:492] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: No bootstrap required, opened a new log
I20260806 09:10:36.663659 6618 ts_tablet_manager.cc:1397] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 09:10:36.669960 6616 raft_consensus.cc:348] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.670328 6616 raft_consensus.cc:374] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:36.670465 6616 raft_consensus.cc:729] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1127b0b9f69145a39405c83ee9ea50e4, State: Initialized, Role: FOLLOWER
I20260806 09:10:36.671113 6616 consensus_queue.cc:260] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.672312 6393 heartbeater.cc:503] Master 127.3.69.62:37109 was elected leader, sending a full tablet report...
I20260806 09:10:36.672850 6616 ts_tablet_manager.cc:1428] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260806 09:10:36.672811 6614 raft_consensus.cc:348] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.673219 6614 raft_consensus.cc:374] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:36.673323 6614 raft_consensus.cc:729] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a73751b43f4ae78cea34ef099dbe88, State: Initialized, Role: FOLLOWER
I20260806 09:10:36.673969 6614 consensus_queue.cc:260] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.675153 6592 heartbeater.cc:503] Master 127.3.69.62:37109 was elected leader, sending a full tablet report...
I20260806 09:10:36.675014 6618 raft_consensus.cc:348] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.675472 6618 raft_consensus.cc:374] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:36.675567 6618 raft_consensus.cc:729] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf15e6e8be34145b453ff076325278c, State: Initialized, Role: FOLLOWER
I20260806 09:10:36.675773 6614 ts_tablet_manager.cc:1428] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260806 09:10:36.676170 6618 consensus_queue.cc:260] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.680534 6618 ts_tablet_manager.cc:1428] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260806 09:10:36.692047 6624 raft_consensus.cc:482] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:36.692348 6624 raft_consensus.cc:504] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.693921 6624 leader_election.cc:288] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565), f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649)
I20260806 09:10:36.701216 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.694375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:57707 (local address 127.3.69.2:45565)
0806 09:10:36.694583 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.694587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.695306 (+ 719us) server_negotiation.cc:407] Connection header received
0806 09:10:36.695504 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.695510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.695612 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.695729 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:36.696431 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.697291 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.699391 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.700223 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.700380 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.700498 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.700812 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.700937 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:10:36.701339 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.694323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:45565 (local address 127.3.69.1:57707)
0806 09:10:36.695049 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:36.695099 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:36.695344 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:36.695757 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:36.695761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:36.695798 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:36.696282 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.696293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.697419 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.697427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.699239 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.699252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.700447 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.700454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.700644 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.700669 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.700776 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.700903 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":63,"thread_start_us":414,"threads_started":1}
I20260806 09:10:36.701359 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.695491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:34285 (local address 127.3.69.3:45649)
0806 09:10:36.695699 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.695706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.696070 (+ 364us) server_negotiation.cc:407] Connection header received
0806 09:10:36.696242 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.696247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.696311 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.696405 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:36.696932 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.697605 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.699577 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.700430 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.700579 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.700692 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.700965 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.701083 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 09:10:36.701936 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.695272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:45649 (local address 127.3.69.1:34285)
0806 09:10:36.695934 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:36.695952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:36.696096 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:36.696420 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:36.696424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:36.696447 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:36.696773 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.696781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.697764 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.697770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.699430 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.699452 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.700652 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.700658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.700813 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.700835 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.700940 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.701067 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":433,"threads_started":1}
I20260806 09:10:36.702548 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612f1cc160ec4bf2b06266535d2c0c3c" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4" is_pre_election: true
I20260806 09:10:36.702987 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612f1cc160ec4bf2b06266535d2c0c3c" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a73751b43f4ae78cea34ef099dbe88" is_pre_election: true
I20260806 09:10:36.703169 6333 raft_consensus.cc:2454] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf15e6e8be34145b453ff076325278c in term 0.
I20260806 09:10:36.703544 6523 raft_consensus.cc:2454] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf15e6e8be34145b453ff076325278c in term 0.
I20260806 09:10:36.703899 6037 leader_election.cc:302] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, 2bf15e6e8be34145b453ff076325278c; no voters:
I20260806 09:10:36.704331 6624 raft_consensus.cc:2781] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:36.704483 6624 raft_consensus.cc:482] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:36.704593 6624 raft_consensus.cc:3037] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:36.707114 6624 raft_consensus.cc:504] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.707710 6624 leader_election.cc:288] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 election: Requested vote from peers 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565), f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649)
I20260806 09:10:36.708156 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612f1cc160ec4bf2b06266535d2c0c3c" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4"
I20260806 09:10:36.708227 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612f1cc160ec4bf2b06266535d2c0c3c" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a73751b43f4ae78cea34ef099dbe88"
I20260806 09:10:36.708348 6333 raft_consensus.cc:3037] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:36.708484 6523 raft_consensus.cc:3037] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:36.710610 6333 raft_consensus.cc:2454] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf15e6e8be34145b453ff076325278c in term 1.
I20260806 09:10:36.710741 6523 raft_consensus.cc:2454] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf15e6e8be34145b453ff076325278c in term 1.
I20260806 09:10:36.711005 6037 leader_election.cc:302] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, 2bf15e6e8be34145b453ff076325278c; no voters:
I20260806 09:10:36.711330 6624 raft_consensus.cc:2781] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:36.711781 6624 raft_consensus.cc:686] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 1 LEADER]: Becoming Leader. State: Replica: 2bf15e6e8be34145b453ff076325278c, State: Running, Role: LEADER
I20260806 09:10:36.712276 6624 consensus_queue.cc:237] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.716542 5916 catalog_manager.cc:5495] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bf15e6e8be34145b453ff076325278c (127.3.69.1). New cstate: current_term: 1 leader_uuid: "2bf15e6e8be34145b453ff076325278c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } health_report { overall_health: UNKNOWN } } }
W20260806 09:10:36.738783 6209 tablet.cc:2369] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:10:36.759297 6394 tablet.cc:2369] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 09:10:36 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007434, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:36.773656 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.758141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37109 (local address 127.0.0.1:48026)
0806 09:10:36.759220 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:36.759249 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:36.759428 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:36.759976 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:36.759982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:36.761012 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:36.761442 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.761459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.762769 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.762777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.764164 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:36.764180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.765267 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.765276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:36.765506 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.766306 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:36.766333 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:36.768750 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:36.771400 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:36.771410 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:36.771416 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:36.771856 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:36.772192 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:36.772195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:36.772198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:36.772346 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:36.772828 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:36.772834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:36.772983 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 09:10:36.773112 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 09:10:36.773309 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":707,"threads_started":1}
I20260806 09:10:36.773792 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:36.757959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.3.69.62:37109)
0806 09:10:36.758230 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:36.758239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:36.759378 (+ 1139us) server_negotiation.cc:407] Connection header received
0806 09:10:36.759581 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:36.759588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:36.759669 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:36.759746 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:36.761614 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.762644 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.764346 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:36.765050 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:36.765216 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:36.768920 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:36.768925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:36.768928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:36.768957 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:36.771279 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.771970 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.771974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.771976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.772098 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:36.772443 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:36.772446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:36.772448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:36.772703 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:36.772811 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:36.773261 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 09:10:36.773442 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:10:36.778745 5916 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:48026:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:10:36.779841 5916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:36.786667 6115 tablet_service.cc:1467] Processing CreateTablet for tablet 816acc5826a74a03ae4c9ab65c21b614 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ecd5646530d452da4d233443b9704e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:10:36.786834 6298 tablet_service.cc:1467] Processing CreateTablet for tablet 816acc5826a74a03ae4c9ab65c21b614 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ecd5646530d452da4d233443b9704e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:10:36.787783 6115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816acc5826a74a03ae4c9ab65c21b614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:36.787783 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816acc5826a74a03ae4c9ab65c21b614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:36.787274 6497 tablet_service.cc:1467] Processing CreateTablet for tablet 816acc5826a74a03ae4c9ab65c21b614 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ecd5646530d452da4d233443b9704e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:10:36.788125 6497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816acc5826a74a03ae4c9ab65c21b614. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:36.793857 6614 tablet_bootstrap.cc:492] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: Bootstrap starting.
W20260806 09:10:36.794430 6593 tablet.cc:2369] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:10:36.795053 6618 tablet_bootstrap.cc:492] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: Bootstrap starting.
I20260806 09:10:36.795472 6614 tablet_bootstrap.cc:654] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:36.796558 6616 tablet_bootstrap.cc:492] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: Bootstrap starting.
I20260806 09:10:36.796845 6618 tablet_bootstrap.cc:654] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:36.798197 6614 tablet_bootstrap.cc:492] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: No bootstrap required, opened a new log
I20260806 09:10:36.798388 6614 ts_tablet_manager.cc:1397] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 09:10:36.798455 6616 tablet_bootstrap.cc:654] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:36.799105 6618 tablet_bootstrap.cc:492] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: No bootstrap required, opened a new log
I20260806 09:10:36.799259 6618 ts_tablet_manager.cc:1397] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260806 09:10:36.799245 6614 raft_consensus.cc:348] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.799531 6614 raft_consensus.cc:374] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:36.799626 6614 raft_consensus.cc:729] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a73751b43f4ae78cea34ef099dbe88, State: Initialized, Role: FOLLOWER
I20260806 09:10:36.799927 6614 consensus_queue.cc:260] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.800086 6618 raft_consensus.cc:348] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.800308 6618 raft_consensus.cc:374] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:36.800417 6618 raft_consensus.cc:729] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf15e6e8be34145b453ff076325278c, State: Initialized, Role: FOLLOWER
I20260806 09:10:36.800665 6618 consensus_queue.cc:260] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.800601 6623 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:36.800937 6623 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:36.801168 6624 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:36.801340 6618 ts_tablet_manager.cc:1428] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 09:10:36.801458 6624 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:36.802328 6614 ts_tablet_manager.cc:1428] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 09:10:36.804759 6616 tablet_bootstrap.cc:492] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: No bootstrap required, opened a new log
I20260806 09:10:36.804930 6616 ts_tablet_manager.cc:1397] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 09:10:36.805787 6616 raft_consensus.cc:348] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.806003 6616 raft_consensus.cc:374] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:36.806106 6616 raft_consensus.cc:729] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1127b0b9f69145a39405c83ee9ea50e4, State: Initialized, Role: FOLLOWER
I20260806 09:10:36.806365 6616 consensus_queue.cc:260] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:36.806994 6616 ts_tablet_manager.cc:1428] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 09:10:36.806823 6622 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:36.807258 6622 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:37.096067 6623 raft_consensus.cc:482] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:37.096294 6623 raft_consensus.cc:504] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.097934 6623 leader_election.cc:288] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777), 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565)
I20260806 09:10:37.104209 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.098430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:60373 (local address 127.3.69.1:34777)
0806 09:10:37.098668 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:37.098704 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:37.099256 (+ 552us) server_negotiation.cc:407] Connection header received
0806 09:10:37.099469 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:37.099473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:37.099548 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:37.099659 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:37.100059 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.100770 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.102426 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.103227 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.103365 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.103471 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:37.103815 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 09:10:37.103937 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:10:37.104326 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.098097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34777 (local address 127.3.69.3:60373)
0806 09:10:37.099129 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:37.099147 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:37.099322 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:37.099673 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:37.099678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:37.099701 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:37.099917 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.099926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.100916 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.100922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.102263 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.102278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.103398 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.103404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.103585 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.103663 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 09:10:37.103790 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 09:10:37.103937 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":22,"thread_start_us":758,"threads_started":1}
I20260806 09:10:37.104490 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.098389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:54381 (local address 127.3.69.2:45565)
0806 09:10:37.098681 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:37.098690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:37.099349 (+ 659us) server_negotiation.cc:407] Connection header received
0806 09:10:37.099591 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:37.099599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:37.099695 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:37.099795 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:37.100284 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.101050 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.102705 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.103518 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.103668 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.103769 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:37.104083 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 09:10:37.104217 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:10:37.104908 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.098283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:45565 (local address 127.3.69.3:54381)
0806 09:10:37.099251 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:37.099264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:37.099402 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:37.099812 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:37.099816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:37.099839 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:37.100137 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.100145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.101185 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.101190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.102555 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.102566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.103766 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.103773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.103927 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.103949 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 09:10:37.104055 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 09:10:37.104176 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":7,"thread_start_us":579,"threads_started":1}
I20260806 09:10:37.105315 6148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816acc5826a74a03ae4c9ab65c21b614" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf15e6e8be34145b453ff076325278c" is_pre_election: true
I20260806 09:10:37.105584 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816acc5826a74a03ae4c9ab65c21b614" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4" is_pre_election: true
I20260806 09:10:37.105788 6148 raft_consensus.cc:2454] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 0.
I20260806 09:10:37.105859 6333 raft_consensus.cc:2454] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 0.
I20260806 09:10:37.106312 6421 leader_election.cc:302] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, f1a73751b43f4ae78cea34ef099dbe88; no voters:
I20260806 09:10:37.106757 6623 raft_consensus.cc:2781] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:37.106890 6623 raft_consensus.cc:482] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:37.107000 6623 raft_consensus.cc:3037] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:37.109109 6623 raft_consensus.cc:504] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.109642 6623 leader_election.cc:288] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 election: Requested vote from peers 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777), 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565)
I20260806 09:10:37.110096 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816acc5826a74a03ae4c9ab65c21b614" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4"
I20260806 09:10:37.110116 6148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816acc5826a74a03ae4c9ab65c21b614" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf15e6e8be34145b453ff076325278c"
I20260806 09:10:37.110302 6148 raft_consensus.cc:3037] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:37.110321 6333 raft_consensus.cc:3037] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:37.113351 6333 raft_consensus.cc:2454] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 1.
I20260806 09:10:37.113366 6148 raft_consensus.cc:2454] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 1.
I20260806 09:10:37.113802 6420 leader_election.cc:302] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, 2bf15e6e8be34145b453ff076325278c, f1a73751b43f4ae78cea34ef099dbe88; no voters:
I20260806 09:10:37.114097 6623 raft_consensus.cc:2781] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:37.114521 6623 raft_consensus.cc:686] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 LEADER]: Becoming Leader. State: Replica: f1a73751b43f4ae78cea34ef099dbe88, State: Running, Role: LEADER
I20260806 09:10:37.114990 6623 consensus_queue.cc:237] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.119259 6651 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: TxnStatusTablet state changed. Reason: New leader f1a73751b43f4ae78cea34ef099dbe88. Latest consensus state: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:37.119632 6651 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:10:37.120357 6655 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:10:37.120613 6655 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:10:37.121008 5915 catalog_manager.cc:5495] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3). New cstate: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } health_report { overall_health: HEALTHY } } }
I20260806 09:10:37.135673 6624 consensus_queue.cc:1035] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:37.143018 6624 consensus_queue.cc:1035] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:37.175757 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.169169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:45649 (local address 127.0.0.1:55292)
0806 09:10:37.169401 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:37.169420 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:37.169542 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:37.170085 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:37.170088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:37.170098 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:10:37.170255 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.170264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.171841 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.171845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.172814 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.172826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.173801 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.173806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.173963 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.174017 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.175151 (+ 1134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.175173 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 09:10:37.175262 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 09:10:37.175423 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 09:10:37.176347 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.169330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.3.69.3:45649)
0806 09:10:37.169498 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:37.169504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:37.169525 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:10:37.169763 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:37.169769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:37.169899 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:37.170055 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:37.170390 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.171692 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.172961 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.173619 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.173787 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.174202 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.174921 (+ 719us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.175063 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:37.175369 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 09:10:37.175511 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:10:37.182955 6333 raft_consensus.cc:1261] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Refusing update from remote peer f1a73751b43f4ae78cea34ef099dbe88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:37.183291 6148 raft_consensus.cc:1261] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Refusing update from remote peer f1a73751b43f4ae78cea34ef099dbe88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:37.183686 6651 consensus_queue.cc:1035] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:37.184125 6623 consensus_queue.cc:1035] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:37.188714 6622 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: TxnStatusTablet state changed. Reason: New leader f1a73751b43f4ae78cea34ef099dbe88. Latest consensus state: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:37.188999 6622 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:37.191461 6663 mvcc.cc:205] Tried to move back new op lower bound from 7315486462693855232 to 7315486462428069888. Current Snapshot: MvccSnapshot[applied={T|T < 7315486462693855232}]
I20260806 09:10:37.192613 6622 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:37.192852 6622 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P 1127b0b9f69145a39405c83ee9ea50e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:37.199200 6667 mvcc.cc:205] Tried to move back new op lower bound from 7315486462693855232 to 7315486462428069888. Current Snapshot: MvccSnapshot[applied={T|T < 7315486462693855232 or (T in {7315486462693855232})}]
I20260806 09:10:37.200428 6628 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: TxnStatusTablet state changed. Reason: New leader f1a73751b43f4ae78cea34ef099dbe88. Latest consensus state: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:37.200778 6628 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:37.207923 6651 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:37.208315 6651 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:10:37.208634 6651 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:37.208997 6651 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P f1a73751b43f4ae78cea34ef099dbe88: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:10:37.209599 6628 tablet_replica.cc:439] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } } }
I20260806 09:10:37.209834 6628 tablet_replica.cc:442] T 816acc5826a74a03ae4c9ab65c21b614 P 2bf15e6e8be34145b453ff076325278c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:10:37.224661 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.218426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.3.69.1:34777)
0806 09:10:37.218579 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:37.218584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:37.219114 (+ 530us) server_negotiation.cc:407] Connection header received
0806 09:10:37.219376 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:37.219379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:37.219463 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:37.219615 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:37.220059 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.220888 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.222408 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.222825 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.222961 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.223451 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.223995 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.224150 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:37.224284 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 09:10:37.224366 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 09:10:37.240520 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.235041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:54259 (local address 127.3.69.3:45649)
0806 09:10:37.235263 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:37.235271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:37.235763 (+ 492us) server_negotiation.cc:407] Connection header received
0806 09:10:37.235973 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:37.235977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:37.236072 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:37.236194 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:37.236628 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.237368 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.238708 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.239213 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.239346 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.239788 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.239883 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.239981 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:37.240132 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 09:10:37.240213 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 09:10:37.240621 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.235024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:45649 (local address 127.3.69.1:54259)
0806 09:10:37.235660 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:37.235675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:37.235832 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:37.236213 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:37.236217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:37.236229 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:10:37.236468 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.236476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.237502 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.237510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.238559 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.238570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.239389 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.239398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.239526 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.239575 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.239983 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.240005 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 09:10:37.240102 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:10:37.240228 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":358,"threads_started":1}
I20260806 09:10:37.280874 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.273206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:50881 (local address 127.3.69.1:34777)
0806 09:10:37.273384 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:37.273390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:37.274392 (+ 1002us) server_negotiation.cc:407] Connection header received
0806 09:10:37.274676 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:37.274681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:37.274756 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:37.274853 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:37.275558 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.276603 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.278722 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.279422 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.279569 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.280074 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.280221 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.280293 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:37.280505 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 09:10:37.280598 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:10:37.281032 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.273249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34777 (local address 127.3.69.3:50881)
0806 09:10:37.274074 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:37.274310 (+ 236us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:37.274521 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:37.274869 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:37.274874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:37.274885 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:10:37.275345 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.275359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.276751 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.276757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.278566 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.278587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.279614 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.279622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.279807 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.279884 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.280338 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.280352 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 09:10:37.280438 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 09:10:37.280608 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":432,"threads_started":1}
I20260806 09:10:37.322180 5916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:37.322494 5916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:37.326525 5916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: Sending DeleteTablet for 3 replicas of tablet 612f1cc160ec4bf2b06266535d2c0c3c
I20260806 09:10:37.327718 6115 tablet_service.cc:1514] Processing DeleteTablet for tablet 612f1cc160ec4bf2b06266535d2c0c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:37 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:41096
I20260806 09:10:37.327700 6298 tablet_service.cc:1514] Processing DeleteTablet for tablet 612f1cc160ec4bf2b06266535d2c0c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:37 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:45964
I20260806 09:10:37.330507 6497 tablet_service.cc:1514] Processing DeleteTablet for tablet 612f1cc160ec4bf2b06266535d2c0c3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:37 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:55276
I20260806 09:10:37.336021 5916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:10:37.336750 5916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:37.339659 6693 tablet_replica.cc:330] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: stopping tablet replica
I20260806 09:10:37.340238 6693 raft_consensus.cc:2229] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:37.340617 6693 raft_consensus.cc:2258] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:37.343109 6693 ts_tablet_manager.cc:1905] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:37.344626 6297 tablet_service.cc:1467] Processing CreateTablet for tablet ef0a5122a2404de9826d1d4fd3af341a (DEFAULT_TABLE table=test-table [id=ded7463d521d45f4b87ead61ea08e337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:37.345103 6297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0a5122a2404de9826d1d4fd3af341a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:37.348291 6692 tablet_replica.cc:330] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: stopping tablet replica
I20260806 09:10:37.348285 6497 tablet_service.cc:1467] Processing CreateTablet for tablet ef0a5122a2404de9826d1d4fd3af341a (DEFAULT_TABLE table=test-table [id=ded7463d521d45f4b87ead61ea08e337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:37.348892 6692 raft_consensus.cc:2229] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:37.349375 6112 tablet_service.cc:1467] Processing CreateTablet for tablet ef0a5122a2404de9826d1d4fd3af341a (DEFAULT_TABLE table=test-table [id=ded7463d521d45f4b87ead61ea08e337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:37.349658 6692 raft_consensus.cc:2258] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:37.349954 6112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0a5122a2404de9826d1d4fd3af341a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:37.351800 6692 ts_tablet_manager.cc:1905] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:37.358999 6692 ts_tablet_manager.cc:1918] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:37.359244 6692 log.cc:1196] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/612f1cc160ec4bf2b06266535d2c0c3c
I20260806 09:10:37.360255 6692 ts_tablet_manager.cc:1939] T 612f1cc160ec4bf2b06266535d2c0c3c P 1127b0b9f69145a39405c83ee9ea50e4: Deleting consensus metadata
I20260806 09:10:37.362182 5891 catalog_manager.cc:4841] TS 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565): tablet 612f1cc160ec4bf2b06266535d2c0c3c (table test-table [id=4bf587808d9a4659b107d6ea0ab031e0]) successfully deleted
I20260806 09:10:37.363296 6695 tablet_bootstrap.cc:492] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: Bootstrap starting.
I20260806 09:10:37.364014 6497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0a5122a2404de9826d1d4fd3af341a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:37.365734 6695 tablet_bootstrap.cc:654] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:37.373394 6691 tablet_replica.cc:330] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: stopping tablet replica
I20260806 09:10:37.373879 6691 raft_consensus.cc:2229] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:10:37.374035 6694 tablet_bootstrap.cc:492] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: Bootstrap starting.
I20260806 09:10:37.374516 6691 raft_consensus.cc:2258] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:37.376617 6694 tablet_bootstrap.cc:654] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:37.376866 6691 ts_tablet_manager.cc:1905] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:37.382170 6697 tablet_bootstrap.cc:492] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: Bootstrap starting.
I20260806 09:10:37.382431 6695 tablet_bootstrap.cc:492] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: No bootstrap required, opened a new log
I20260806 09:10:37.382591 6695 ts_tablet_manager.cc:1397] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.010s
I20260806 09:10:37.383587 6695 raft_consensus.cc:348] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.383893 6695 raft_consensus.cc:374] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:37.384006 6695 raft_consensus.cc:729] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf15e6e8be34145b453ff076325278c, State: Initialized, Role: FOLLOWER
I20260806 09:10:37.384260 6695 consensus_queue.cc:260] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.384985 6695 ts_tablet_manager.cc:1428] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 09:10:37.385643 6691 ts_tablet_manager.cc:1918] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:37.385811 6691 log.cc:1196] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/612f1cc160ec4bf2b06266535d2c0c3c
I20260806 09:10:37.385954 6697 tablet_bootstrap.cc:654] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:37.386375 6691 ts_tablet_manager.cc:1939] T 612f1cc160ec4bf2b06266535d2c0c3c P 2bf15e6e8be34145b453ff076325278c: Deleting consensus metadata
I20260806 09:10:37.388025 5890 catalog_manager.cc:4841] TS 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777): tablet 612f1cc160ec4bf2b06266535d2c0c3c (table test-table [id=4bf587808d9a4659b107d6ea0ab031e0]) successfully deleted
I20260806 09:10:37.389694 6665 raft_consensus.cc:482] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:37.389860 6665 raft_consensus.cc:504] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.390745 6665 leader_election.cc:288] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565), f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649)
I20260806 09:10:37.391218 6694 tablet_bootstrap.cc:492] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: No bootstrap required, opened a new log
I20260806 09:10:37.391386 6694 ts_tablet_manager.cc:1397] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20260806 09:10:37.391317 6330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a5122a2404de9826d1d4fd3af341a" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4" is_pre_election: true
I20260806 09:10:37.392184 6694 raft_consensus.cc:348] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.392409 6694 raft_consensus.cc:374] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:37.392510 6694 raft_consensus.cc:729] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1127b0b9f69145a39405c83ee9ea50e4, State: Initialized, Role: FOLLOWER
W20260806 09:10:37.392571 6037 leader_election.cc:341] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565): Illegal state: must be running to vote when last-logged opid is not known
I20260806 09:10:37.392752 6693 ts_tablet_manager.cc:1918] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:37.392939 6693 log.cc:1196] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/612f1cc160ec4bf2b06266535d2c0c3c
I20260806 09:10:37.393179 6694 consensus_queue.cc:260] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.393831 6694 ts_tablet_manager.cc:1428] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 09:10:37.393150 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a5122a2404de9826d1d4fd3af341a" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a73751b43f4ae78cea34ef099dbe88" is_pre_election: true
I20260806 09:10:37.396793 6693 ts_tablet_manager.cc:1939] T 612f1cc160ec4bf2b06266535d2c0c3c P f1a73751b43f4ae78cea34ef099dbe88: Deleting consensus metadata
I20260806 09:10:37.396981 6697 tablet_bootstrap.cc:492] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: No bootstrap required, opened a new log
I20260806 09:10:37.397152 6697 ts_tablet_manager.cc:1397] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
W20260806 09:10:37.397197 6037 leader_election.cc:341] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649): Illegal state: must be running to vote when last-logged opid is not known
I20260806 09:10:37.397337 6037 leader_election.cc:302] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bf15e6e8be34145b453ff076325278c; no voters: 1127b0b9f69145a39405c83ee9ea50e4, f1a73751b43f4ae78cea34ef099dbe88
I20260806 09:10:37.398048 6697 raft_consensus.cc:348] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.398330 6697 raft_consensus.cc:374] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:37.398450 6697 raft_consensus.cc:729] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a73751b43f4ae78cea34ef099dbe88, State: Initialized, Role: FOLLOWER
I20260806 09:10:37.398731 6697 consensus_queue.cc:260] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.398968 6665 raft_consensus.cc:2726] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 09:10:37.399353 6697 ts_tablet_manager.cc:1428] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 09:10:37.400223 5891 catalog_manager.cc:4841] TS f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649): tablet 612f1cc160ec4bf2b06266535d2c0c3c (table test-table [id=4bf587808d9a4659b107d6ea0ab031e0]) successfully deleted
I20260806 09:10:37.632869 6623 raft_consensus.cc:482] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:37.633081 6623 raft_consensus.cc:504] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.633957 6623 leader_election.cc:288] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777), 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565)
I20260806 09:10:37.634624 6330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a5122a2404de9826d1d4fd3af341a" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4" is_pre_election: true
I20260806 09:10:37.634711 6148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a5122a2404de9826d1d4fd3af341a" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf15e6e8be34145b453ff076325278c" is_pre_election: true
I20260806 09:10:37.634913 6330 raft_consensus.cc:2454] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 0.
I20260806 09:10:37.635015 6148 raft_consensus.cc:2454] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 0.
I20260806 09:10:37.635464 6421 leader_election.cc:302] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, f1a73751b43f4ae78cea34ef099dbe88; no voters:
I20260806 09:10:37.635758 6623 raft_consensus.cc:2781] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:37.635893 6623 raft_consensus.cc:482] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:37.636004 6623 raft_consensus.cc:3037] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:37.638291 6623 raft_consensus.cc:504] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.638888 6623 leader_election.cc:288] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 election: Requested vote from peers 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777), 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565)
I20260806 09:10:37.639467 6148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a5122a2404de9826d1d4fd3af341a" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf15e6e8be34145b453ff076325278c"
I20260806 09:10:37.639490 6330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0a5122a2404de9826d1d4fd3af341a" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4"
I20260806 09:10:37.639703 6148 raft_consensus.cc:3037] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:37.639717 6330 raft_consensus.cc:3037] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:37.642171 6148 raft_consensus.cc:2454] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 1.
I20260806 09:10:37.642653 6420 leader_election.cc:302] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf15e6e8be34145b453ff076325278c, f1a73751b43f4ae78cea34ef099dbe88; no voters:
I20260806 09:10:37.642864 6330 raft_consensus.cc:2454] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 1.
I20260806 09:10:37.642957 6623 raft_consensus.cc:2781] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:37.643201 6623 raft_consensus.cc:686] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 1 LEADER]: Becoming Leader. State: Replica: f1a73751b43f4ae78cea34ef099dbe88, State: Running, Role: LEADER
I20260806 09:10:37.643527 6623 consensus_queue.cc:237] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:37.646595 5915 catalog_manager.cc:5495] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3). New cstate: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } health_report { overall_health: HEALTHY } } }
I20260806 09:10:37.670614 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.665329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.3.69.3:45649)
0806 09:10:37.665563 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:37.665571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:37.665733 (+ 162us) server_negotiation.cc:407] Connection header received
0806 09:10:37.665931 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:37.665941 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:37.666005 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:37.666101 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:37.666475 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.667262 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.668633 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.669294 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.669444 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.669854 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.669984 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.670089 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:37.670244 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 09:10:37.670321 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 09:10:37.684512 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.678666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:39275 (local address 127.3.69.3:45649)
0806 09:10:37.678846 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:37.678852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:37.679037 (+ 185us) server_negotiation.cc:407] Connection header received
0806 09:10:37.679256 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:37.679263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:37.679333 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:37.679405 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:37.679839 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.680623 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.682666 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.683143 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.683244 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.683721 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.683815 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.683885 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:37.684103 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 09:10:37.684224 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 09:10:37.684571 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:37.678718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:45649 (local address 127.3.69.3:39275)
0806 09:10:37.678940 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:37.678961 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:37.679133 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:37.679457 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:37.679462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:37.679475 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:10:37.679720 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.679735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.680791 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.680797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.682525 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:37.682541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:37.683307 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:37.683315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:37.683503 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:37.683559 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:10:37.683926 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:10:37.683938 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:10:37.684037 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:10:37.684212 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 09:10:37.693552 6148 raft_consensus.cc:1261] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Refusing update from remote peer f1a73751b43f4ae78cea34ef099dbe88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:37.693696 6330 raft_consensus.cc:1261] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Refusing update from remote peer f1a73751b43f4ae78cea34ef099dbe88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:37.694250 6686 consensus_queue.cc:1035] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:37.694840 6686 consensus_queue.cc:1035] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:10:37.722548 6663 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 09:10:39.323688 5916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:39.323967 5916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:39.328320 5916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: Sending DeleteTablet for 3 replicas of tablet ef0a5122a2404de9826d1d4fd3af341a
I20260806 09:10:39.329196 6497 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0a5122a2404de9826d1d4fd3af341a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:39 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:55276
I20260806 09:10:39.329205 6115 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0a5122a2404de9826d1d4fd3af341a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:39 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:41096
I20260806 09:10:39.329221 6297 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0a5122a2404de9826d1d4fd3af341a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:39 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:45964
I20260806 09:10:39.341598 6737 tablet_replica.cc:330] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: stopping tablet replica
I20260806 09:10:39.341847 6735 tablet_replica.cc:330] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: stopping tablet replica
I20260806 09:10:39.341976 6737 raft_consensus.cc:2229] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:10:39.342121 6735 raft_consensus.cc:2229] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:39.342450 6735 raft_consensus.cc:2258] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:39.342675 6737 raft_consensus.cc:2258] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:39.343254 6738 tablet_replica.cc:330] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: stopping tablet replica
I20260806 09:10:39.343554 6738 raft_consensus.cc:2229] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:39.343609 6735 ts_tablet_manager.cc:1905] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:39.343698 6737 ts_tablet_manager.cc:1905] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:39.343876 6738 raft_consensus.cc:2258] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:39.344880 6738 ts_tablet_manager.cc:1905] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:39.351155 6735 ts_tablet_manager.cc:1918] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:10:39.351181 6737 ts_tablet_manager.cc:1918] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:10:39.351214 6738 ts_tablet_manager.cc:1918] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:10:39.351316 6737 log.cc:1196] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/ef0a5122a2404de9826d1d4fd3af341a
I20260806 09:10:39.351431 6735 log.cc:1196] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/ef0a5122a2404de9826d1d4fd3af341a
I20260806 09:10:39.351446 6738 log.cc:1196] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/ef0a5122a2404de9826d1d4fd3af341a
I20260806 09:10:39.351910 6738 ts_tablet_manager.cc:1939] T ef0a5122a2404de9826d1d4fd3af341a P 2bf15e6e8be34145b453ff076325278c: Deleting consensus metadata
I20260806 09:10:39.351920 6737 ts_tablet_manager.cc:1939] T ef0a5122a2404de9826d1d4fd3af341a P f1a73751b43f4ae78cea34ef099dbe88: Deleting consensus metadata
I20260806 09:10:39.351948 6735 ts_tablet_manager.cc:1939] T ef0a5122a2404de9826d1d4fd3af341a P 1127b0b9f69145a39405c83ee9ea50e4: Deleting consensus metadata
I20260806 09:10:39.353077 5891 catalog_manager.cc:4841] TS f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649): tablet ef0a5122a2404de9826d1d4fd3af341a (table test-table [id=ded7463d521d45f4b87ead61ea08e337]) successfully deleted
I20260806 09:10:39.353371 5891 catalog_manager.cc:4841] TS 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565): tablet ef0a5122a2404de9826d1d4fd3af341a (table test-table [id=ded7463d521d45f4b87ead61ea08e337]) successfully deleted
I20260806 09:10:39.353811 5890 catalog_manager.cc:4841] TS 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777): tablet ef0a5122a2404de9826d1d4fd3af341a (table test-table [id=ded7463d521d45f4b87ead61ea08e337]) successfully deleted
Aug 06 09:10:39 dist-test-slave-w4v5 krb5kdc[5849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:10:39 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:39.396821 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.382057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.3.69.62:37109)
0806 09:10:39.383161 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:39.383166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:39.383239 (+ 73us) server_negotiation.cc:407] Connection header received
0806 09:10:39.383426 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:39.383430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:39.383495 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:39.383594 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:39.385016 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.385849 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.387155 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.387557 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.387662 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.391822 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:39.391827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:39.391830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:39.391864 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:39.394382 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:39.395018 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:39.395023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:39.395026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:39.395215 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:39.395556 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:39.395560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:39.395562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:39.395839 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:39.396103 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:39.396270 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 09:10:39.396434 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":637,"threads_started":1}
I20260806 09:10:39.400686 5916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:10:39.401474 5916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:39.408459 6297 tablet_service.cc:1467] Processing CreateTablet for tablet 395208945faa4002b39b0dee3cb09a86 (DEFAULT_TABLE table=test-table [id=6dfdbedbb17448c5990d13226dd9b216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:39.408440 6115 tablet_service.cc:1467] Processing CreateTablet for tablet 395208945faa4002b39b0dee3cb09a86 (DEFAULT_TABLE table=test-table [id=6dfdbedbb17448c5990d13226dd9b216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:39.408725 6497 tablet_service.cc:1467] Processing CreateTablet for tablet 395208945faa4002b39b0dee3cb09a86 (DEFAULT_TABLE table=test-table [id=6dfdbedbb17448c5990d13226dd9b216]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:39.408910 6115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395208945faa4002b39b0dee3cb09a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:39.408977 6297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395208945faa4002b39b0dee3cb09a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:39.409145 6497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 395208945faa4002b39b0dee3cb09a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:39.413991 6750 tablet_bootstrap.cc:492] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: Bootstrap starting.
I20260806 09:10:39.415581 6750 tablet_bootstrap.cc:654] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:39.417575 6752 tablet_bootstrap.cc:492] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: Bootstrap starting.
I20260806 09:10:39.418157 6750 tablet_bootstrap.cc:492] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: No bootstrap required, opened a new log
I20260806 09:10:39.418334 6750 ts_tablet_manager.cc:1397] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 09:10:39.419421 6752 tablet_bootstrap.cc:654] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:39.419447 6750 raft_consensus.cc:348] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.419804 6750 raft_consensus.cc:374] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:39.419934 6750 raft_consensus.cc:729] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1127b0b9f69145a39405c83ee9ea50e4, State: Initialized, Role: FOLLOWER
I20260806 09:10:39.420224 6750 consensus_queue.cc:260] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.421202 6750 ts_tablet_manager.cc:1428] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 09:10:39.422461 6752 tablet_bootstrap.cc:492] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: No bootstrap required, opened a new log
I20260806 09:10:39.422622 6752 ts_tablet_manager.cc:1397] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 09:10:39.423475 6752 raft_consensus.cc:348] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.423763 6752 raft_consensus.cc:374] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:39.423870 6752 raft_consensus.cc:729] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a73751b43f4ae78cea34ef099dbe88, State: Initialized, Role: FOLLOWER
I20260806 09:10:39.424094 6752 consensus_queue.cc:260] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.424273 6751 tablet_bootstrap.cc:492] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: Bootstrap starting.
I20260806 09:10:39.424695 6752 ts_tablet_manager.cc:1428] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 09:10:39.425952 6751 tablet_bootstrap.cc:654] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:39.428171 6751 tablet_bootstrap.cc:492] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: No bootstrap required, opened a new log
I20260806 09:10:39.428350 6751 ts_tablet_manager.cc:1397] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 09:10:39.429016 6751 raft_consensus.cc:348] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.429270 6751 raft_consensus.cc:374] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:39.429368 6751 raft_consensus.cc:729] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf15e6e8be34145b453ff076325278c, State: Initialized, Role: FOLLOWER
I20260806 09:10:39.429625 6751 consensus_queue.cc:260] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.430642 6751 ts_tablet_manager.cc:1428] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 09:10:39.676752 6755 raft_consensus.cc:482] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:39.677143 6755 raft_consensus.cc:504] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.678864 6755 leader_election.cc:288] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777), f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649)
I20260806 09:10:39.698746 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.679316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:44407 (local address 127.3.69.1:34777)
0806 09:10:39.691942 (+ 12626us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:39.691951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:39.691981 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:10:39.692079 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:39.692085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:39.692177 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:39.692302 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:39.693158 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.694229 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.696636 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.697508 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.697625 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.697729 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:39.698160 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 09:10:39.698316 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12461,"thread_start_us":12288,"threads_started":1}
I20260806 09:10:39.698781 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.679184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:34777 (local address 127.3.69.2:44407)
0806 09:10:39.691511 (+ 12327us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:39.691537 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:39.691737 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:39.692502 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:39.692507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:39.692542 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:39.692928 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:39.692942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.694387 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.694392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:39.696448 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:39.696461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.697743 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.697751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:39.697936 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.697989 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 09:10:39.698093 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 09:10:39.698247 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12114,"mutex_wait_us":56,"thread_start_us":11927,"threads_started":1}
I20260806 09:10:39.699136 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.679321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:38727 (local address 127.3.69.3:45649)
0806 09:10:39.692596 (+ 13275us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:39.692601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:39.692630 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:10:39.692682 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:39.692685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:39.692742 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:39.692864 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:10:39.693747 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.694794 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.696870 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.697895 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.698034 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.698193 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:39.698589 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 09:10:39.698740 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13124,"thread_start_us":12978,"threads_started":1}
I20260806 09:10:39.699530 6147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395208945faa4002b39b0dee3cb09a86" candidate_uuid: "1127b0b9f69145a39405c83ee9ea50e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf15e6e8be34145b453ff076325278c" is_pre_election: true
I20260806 09:10:39.699697 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.679276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:45649 (local address 127.3.69.2:38727)
0806 09:10:39.692229 (+ 12953us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:39.692249 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:39.692359 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:39.692910 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:39.692915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:39.692943 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:10:39.693509 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:39.693534 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.694975 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.694979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:39.696714 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:39.696731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.698184 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.698190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:39.698386 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.698437 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 09:10:39.698549 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:10:39.698740 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12807,"mutex_wait_us":1,"thread_start_us":12602,"threads_started":1}
I20260806 09:10:39.699848 6147 raft_consensus.cc:2454] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1127b0b9f69145a39405c83ee9ea50e4 in term 0.
I20260806 09:10:39.700423 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395208945faa4002b39b0dee3cb09a86" candidate_uuid: "1127b0b9f69145a39405c83ee9ea50e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a73751b43f4ae78cea34ef099dbe88" is_pre_election: true
I20260806 09:10:39.700454 6233 leader_election.cc:302] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, 2bf15e6e8be34145b453ff076325278c; no voters:
I20260806 09:10:39.700671 6523 raft_consensus.cc:2454] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1127b0b9f69145a39405c83ee9ea50e4 in term 0.
I20260806 09:10:39.700960 6755 raft_consensus.cc:2781] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:39.701131 6755 raft_consensus.cc:482] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:39.701272 6755 raft_consensus.cc:3037] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:39.703658 6755 raft_consensus.cc:504] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.704247 6755 leader_election.cc:288] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [CANDIDATE]: Term 1 election: Requested vote from peers 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777), f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649)
I20260806 09:10:39.704725 6147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395208945faa4002b39b0dee3cb09a86" candidate_uuid: "1127b0b9f69145a39405c83ee9ea50e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf15e6e8be34145b453ff076325278c"
I20260806 09:10:39.704737 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "395208945faa4002b39b0dee3cb09a86" candidate_uuid: "1127b0b9f69145a39405c83ee9ea50e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a73751b43f4ae78cea34ef099dbe88"
I20260806 09:10:39.704931 6147 raft_consensus.cc:3037] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:39.704942 6523 raft_consensus.cc:3037] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:39.706974 6523 raft_consensus.cc:2454] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1127b0b9f69145a39405c83ee9ea50e4 in term 1.
I20260806 09:10:39.707132 6147 raft_consensus.cc:2454] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1127b0b9f69145a39405c83ee9ea50e4 in term 1.
I20260806 09:10:39.707490 6234 leader_election.cc:302] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, f1a73751b43f4ae78cea34ef099dbe88; no voters:
I20260806 09:10:39.707831 6755 raft_consensus.cc:2781] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:39.708300 6755 raft_consensus.cc:686] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 LEADER]: Becoming Leader. State: Replica: 1127b0b9f69145a39405c83ee9ea50e4, State: Running, Role: LEADER
I20260806 09:10:39.708755 6755 consensus_queue.cc:237] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.712252 5916 catalog_manager.cc:5495] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2). New cstate: current_term: 1 leader_uuid: "1127b0b9f69145a39405c83ee9ea50e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } health_report { overall_health: UNKNOWN } } }
I20260806 09:10:39.800980 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.794767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.3.69.2:45565)
0806 09:10:39.795487 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:39.795491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:39.795507 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:10:39.795587 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:39.795590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:39.795664 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:39.795844 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:39.796310 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.797217 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.798523 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.798977 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.799133 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.799649 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:39.800311 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:39.800468 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:39.800575 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 09:10:39.800656 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":495,"threads_started":1}
I20260806 09:10:39.815548 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.810584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:32837 (local address 127.3.69.3:45649)
0806 09:10:39.810820 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:39.810826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:39.811148 (+ 322us) server_negotiation.cc:407] Connection header received
0806 09:10:39.811332 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:39.811336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:39.811402 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:39.811476 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:39.811912 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.812675 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.813980 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.814378 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.814474 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.814876 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:39.814999 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:39.815065 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:39.815276 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 09:10:39.815346 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:10:39.815666 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.810353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:45649 (local address 127.3.69.2:32837)
0806 09:10:39.811021 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:39.811038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:39.811219 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:39.811504 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:39.811507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:39.811516 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:10:39.811794 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:39.811802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.812800 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.812804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:39.813852 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:39.813863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.814527 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.814531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:39.814659 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.814730 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:10:39.815152 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:10:39.815162 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:10:39.815236 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 09:10:39.815343 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":381,"threads_started":1}
I20260806 09:10:39.824970 6147 raft_consensus.cc:1261] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Refusing update from remote peer 1127b0b9f69145a39405c83ee9ea50e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:39.825052 6523 raft_consensus.cc:1261] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Refusing update from remote peer 1127b0b9f69145a39405c83ee9ea50e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:39.825640 6763 consensus_queue.cc:1035] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:39.826278 6755 consensus_queue.cc:1035] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:39.831563 6713 mvcc.cc:205] Tried to move back new op lower bound from 7315486473517121536 to 7315486473051082752. Current Snapshot: MvccSnapshot[applied={T|T < 7315486473517121536}]
I20260806 09:10:39.855985 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.850152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:50761 (local address 127.3.69.2:45565)
0806 09:10:39.850337 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:39.850343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:39.850806 (+ 463us) server_negotiation.cc:407] Connection header received
0806 09:10:39.850926 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:39.850931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:39.851007 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:39.851139 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:39.851659 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.852710 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.854210 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.854628 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.854779 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.855215 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:39.855362 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:39.855467 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:39.855611 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 09:10:39.855696 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:10:39.856040 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:39.850079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:45565 (local address 127.3.69.3:50761)
0806 09:10:39.850709 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:39.850725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:39.850830 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:39.851146 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:39.851150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:39.851162 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:10:39.851484 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:39.851510 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.852849 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.852855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:39.854068 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:39.854080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:39.854814 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:39.854822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:39.854961 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:39.854997 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:10:39.855477 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:10:39.855489 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:10:39.855586 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:10:39.855714 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":359,"threads_started":1}
I20260806 09:10:39.885993 5916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:39.886289 5916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:39.889669 5916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: Sending DeleteTablet for 3 replicas of tablet 395208945faa4002b39b0dee3cb09a86
I20260806 09:10:39.890514 6115 tablet_service.cc:1514] Processing DeleteTablet for tablet 395208945faa4002b39b0dee3cb09a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:39 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:41096
I20260806 09:10:39.890573 6297 tablet_service.cc:1514] Processing DeleteTablet for tablet 395208945faa4002b39b0dee3cb09a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:39 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:45964
I20260806 09:10:39.891386 6497 tablet_service.cc:1514] Processing DeleteTablet for tablet 395208945faa4002b39b0dee3cb09a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:39 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:55276
I20260806 09:10:39.891543 5916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:10:39.892282 5916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:10:39.895483 6786 tablet_replica.cc:330] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: stopping tablet replica
I20260806 09:10:39.895722 6786 raft_consensus.cc:2229] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:39.896005 6786 raft_consensus.cc:2258] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:39.900946 6497 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a571ae98049cfa9e2f57d7eae3170 (DEFAULT_TABLE table=test-table [id=557ebf6b108d4b69bdf70c4f7047132e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:39.900986 6298 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a571ae98049cfa9e2f57d7eae3170 (DEFAULT_TABLE table=test-table [id=557ebf6b108d4b69bdf70c4f7047132e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:39.901473 6497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1a571ae98049cfa9e2f57d7eae3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:39.901604 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1a571ae98049cfa9e2f57d7eae3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:39.902851 6786 ts_tablet_manager.cc:1905] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:39.903748 6112 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a571ae98049cfa9e2f57d7eae3170 (DEFAULT_TABLE table=test-table [id=557ebf6b108d4b69bdf70c4f7047132e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:10:39.904177 6112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1a571ae98049cfa9e2f57d7eae3170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:39.908608 6750 tablet_bootstrap.cc:492] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: Bootstrap starting.
I20260806 09:10:39.908653 6752 tablet_bootstrap.cc:492] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: Bootstrap starting.
I20260806 09:10:39.910223 6750 tablet_bootstrap.cc:654] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:39.910249 6752 tablet_bootstrap.cc:654] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:39.912446 6783 tablet_replica.cc:330] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: stopping tablet replica
I20260806 09:10:39.912636 6783 raft_consensus.cc:2229] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:39.912753 6752 tablet_bootstrap.cc:492] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: No bootstrap required, opened a new log
I20260806 09:10:39.912794 6751 tablet_bootstrap.cc:492] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: Bootstrap starting.
I20260806 09:10:39.912868 6783 raft_consensus.cc:2258] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:39.912909 6752 ts_tablet_manager.cc:1397] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260806 09:10:39.914219 6750 tablet_bootstrap.cc:492] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: No bootstrap required, opened a new log
I20260806 09:10:39.914116 6752 raft_consensus.cc:348] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.914394 6750 ts_tablet_manager.cc:1397] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260806 09:10:39.914440 6751 tablet_bootstrap.cc:654] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:39.914577 6752 raft_consensus.cc:374] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:39.914728 6752 raft_consensus.cc:729] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a73751b43f4ae78cea34ef099dbe88, State: Initialized, Role: FOLLOWER
I20260806 09:10:39.915066 6752 consensus_queue.cc:260] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.915210 6750 raft_consensus.cc:348] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.915450 6750 raft_consensus.cc:374] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:39.916589 6750 raft_consensus.cc:729] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1127b0b9f69145a39405c83ee9ea50e4, State: Initialized, Role: FOLLOWER
I20260806 09:10:39.916875 6750 consensus_queue.cc:260] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.918020 6750 ts_tablet_manager.cc:1428] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:10:39.918246 6752 ts_tablet_manager.cc:1428] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260806 09:10:39.923329 6783 ts_tablet_manager.cc:1905] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:39.926607 6784 tablet_replica.cc:330] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: stopping tablet replica
I20260806 09:10:39.926872 6784 raft_consensus.cc:2229] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:10:39.927656 6784 raft_consensus.cc:2258] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:39.931459 6784 ts_tablet_manager.cc:1905] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:39.932508 6786 ts_tablet_manager.cc:1918] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:39.932650 6786 log.cc:1196] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/395208945faa4002b39b0dee3cb09a86
I20260806 09:10:39.933120 6783 ts_tablet_manager.cc:1918] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:39.933197 6786 ts_tablet_manager.cc:1939] T 395208945faa4002b39b0dee3cb09a86 P f1a73751b43f4ae78cea34ef099dbe88: Deleting consensus metadata
I20260806 09:10:39.933380 6783 log.cc:1196] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/395208945faa4002b39b0dee3cb09a86
I20260806 09:10:39.933911 6783 ts_tablet_manager.cc:1939] T 395208945faa4002b39b0dee3cb09a86 P 2bf15e6e8be34145b453ff076325278c: Deleting consensus metadata
I20260806 09:10:39.934369 5891 catalog_manager.cc:4841] TS f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649): tablet 395208945faa4002b39b0dee3cb09a86 (table test-table [id=6dfdbedbb17448c5990d13226dd9b216]) successfully deleted
I20260806 09:10:39.935667 5890 catalog_manager.cc:4841] TS 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777): tablet 395208945faa4002b39b0dee3cb09a86 (table test-table [id=6dfdbedbb17448c5990d13226dd9b216]) successfully deleted
I20260806 09:10:39.936070 6751 tablet_bootstrap.cc:492] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: No bootstrap required, opened a new log
I20260806 09:10:39.936239 6751 ts_tablet_manager.cc:1397] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.012s
I20260806 09:10:39.937053 6751 raft_consensus.cc:348] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.937314 6751 raft_consensus.cc:374] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:39.937399 6751 raft_consensus.cc:729] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf15e6e8be34145b453ff076325278c, State: Initialized, Role: FOLLOWER
I20260806 09:10:39.937649 6751 consensus_queue.cc:260] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:39.937817 6784 ts_tablet_manager.cc:1918] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:10:39.937963 6784 log.cc:1196] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/395208945faa4002b39b0dee3cb09a86
I20260806 09:10:39.938450 6784 ts_tablet_manager.cc:1939] T 395208945faa4002b39b0dee3cb09a86 P 1127b0b9f69145a39405c83ee9ea50e4: Deleting consensus metadata
I20260806 09:10:39.939436 6751 ts_tablet_manager.cc:1428] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260806 09:10:39.939682 5891 catalog_manager.cc:4841] TS 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565): tablet 395208945faa4002b39b0dee3cb09a86 (table test-table [id=6dfdbedbb17448c5990d13226dd9b216]) successfully deleted
I20260806 09:10:40.148466 6757 raft_consensus.cc:482] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:40.148785 6757 raft_consensus.cc:504] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:40.149458 6686 raft_consensus.cc:482] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:10:40.149834 6757 leader_election.cc:288] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565), f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649)
I20260806 09:10:40.149802 6686 raft_consensus.cc:504] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:40.150580 6686 leader_election.cc:288] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777), 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565)
I20260806 09:10:40.150651 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a571ae98049cfa9e2f57d7eae3170" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a73751b43f4ae78cea34ef099dbe88" is_pre_election: true
I20260806 09:10:40.150852 6523 raft_consensus.cc:2454] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf15e6e8be34145b453ff076325278c in term 0.
I20260806 09:10:40.150738 6330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a571ae98049cfa9e2f57d7eae3170" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4" is_pre_election: true
I20260806 09:10:40.151063 6330 raft_consensus.cc:2454] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf15e6e8be34145b453ff076325278c in term 0.
I20260806 09:10:40.151392 6037 leader_election.cc:302] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf15e6e8be34145b453ff076325278c, f1a73751b43f4ae78cea34ef099dbe88; no voters:
I20260806 09:10:40.151661 6757 raft_consensus.cc:2781] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:40.151829 6757 raft_consensus.cc:482] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:40.151940 6757 raft_consensus.cc:3037] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:40.151929 6330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a571ae98049cfa9e2f57d7eae3170" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4" is_pre_election: true
I20260806 09:10:40.152189 6330 raft_consensus.cc:2454] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 0.
I20260806 09:10:40.152657 6421 leader_election.cc:302] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, f1a73751b43f4ae78cea34ef099dbe88; no voters:
I20260806 09:10:40.152796 6147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a571ae98049cfa9e2f57d7eae3170" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf15e6e8be34145b453ff076325278c" is_pre_election: true
I20260806 09:10:40.152999 6686 raft_consensus.cc:2781] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:10:40.153093 6686 raft_consensus.cc:482] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:10:40.153174 6686 raft_consensus.cc:3037] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:40.155212 6757 raft_consensus.cc:504] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:40.155313 6686 raft_consensus.cc:504] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:40.155814 6147 raft_consensus.cc:2379] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1a73751b43f4ae78cea34ef099dbe88 in current term 1: Already voted for candidate 2bf15e6e8be34145b453ff076325278c in this term.
I20260806 09:10:40.156349 6757 leader_election.cc:288] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 election: Requested vote from peers 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565), f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649)
I20260806 09:10:40.156983 6147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a571ae98049cfa9e2f57d7eae3170" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf15e6e8be34145b453ff076325278c"
I20260806 09:10:40.157042 6333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a571ae98049cfa9e2f57d7eae3170" candidate_uuid: "f1a73751b43f4ae78cea34ef099dbe88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4"
I20260806 09:10:40.157119 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a571ae98049cfa9e2f57d7eae3170" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a73751b43f4ae78cea34ef099dbe88"
I20260806 09:10:40.157248 6147 raft_consensus.cc:2379] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f1a73751b43f4ae78cea34ef099dbe88 in current term 1: Already voted for candidate 2bf15e6e8be34145b453ff076325278c in this term.
I20260806 09:10:40.157258 6333 raft_consensus.cc:3037] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:40.157363 6523 raft_consensus.cc:2379] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2bf15e6e8be34145b453ff076325278c in current term 1: Already voted for candidate f1a73751b43f4ae78cea34ef099dbe88 in this term.
I20260806 09:10:40.157965 6330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a571ae98049cfa9e2f57d7eae3170" candidate_uuid: "2bf15e6e8be34145b453ff076325278c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1127b0b9f69145a39405c83ee9ea50e4"
I20260806 09:10:40.158492 6686 leader_election.cc:288] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 election: Requested vote from peers 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777), 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565)
I20260806 09:10:40.158677 6037 leader_election.cc:302] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bf15e6e8be34145b453ff076325278c; no voters: 1127b0b9f69145a39405c83ee9ea50e4, f1a73751b43f4ae78cea34ef099dbe88
I20260806 09:10:40.158964 6757 raft_consensus.cc:2726] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 09:10:40.159911 6333 raft_consensus.cc:2454] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a73751b43f4ae78cea34ef099dbe88 in term 1.
I20260806 09:10:40.160346 6421 leader_election.cc:302] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1127b0b9f69145a39405c83ee9ea50e4, f1a73751b43f4ae78cea34ef099dbe88; no voters: 2bf15e6e8be34145b453ff076325278c
I20260806 09:10:40.160656 6686 raft_consensus.cc:2781] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:40.160821 6686 raft_consensus.cc:686] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 LEADER]: Becoming Leader. State: Replica: f1a73751b43f4ae78cea34ef099dbe88, State: Running, Role: LEADER
I20260806 09:10:40.161100 6686 consensus_queue.cc:237] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } }
I20260806 09:10:40.164004 5915 catalog_manager.cc:5495] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3). New cstate: current_term: 1 leader_uuid: "f1a73751b43f4ae78cea34ef099dbe88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a73751b43f4ae78cea34ef099dbe88" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 45649 } health_report { overall_health: HEALTHY } } }
I20260806 09:10:40.221050 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:40.215983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.3.69.3:45649)
0806 09:10:40.216157 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:40.216164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:40.216275 (+ 111us) server_negotiation.cc:407] Connection header received
0806 09:10:40.216493 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:40.216503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:40.216566 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:40.216694 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:10:40.217092 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:40.217860 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:40.219216 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:40.219635 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:40.219781 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:40.220232 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:10:40.220374 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:10:40.220461 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:40.220617 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 09:10:40.220706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 09:10:40.230633 6333 raft_consensus.cc:1261] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Refusing update from remote peer f1a73751b43f4ae78cea34ef099dbe88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:40.230727 6147 raft_consensus.cc:1261] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Refusing update from remote peer f1a73751b43f4ae78cea34ef099dbe88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:10:40.231299 6718 consensus_queue.cc:1035] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1127b0b9f69145a39405c83ee9ea50e4" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:10:40.231653 6686 consensus_queue.cc:1035] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf15e6e8be34145b453ff076325278c" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:10:40.259681 6710 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 09:10:41.874266 5916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:41.874481 5916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:10:41.878708 5916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b21887c166da4205a4fc5b4e32a19ba8: Sending DeleteTablet for 3 replicas of tablet 2a1a571ae98049cfa9e2f57d7eae3170
I20260806 09:10:41.879621 6112 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a571ae98049cfa9e2f57d7eae3170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:41 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:41096
I20260806 09:10:41.879761 6297 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a571ae98049cfa9e2f57d7eae3170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:41 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:45964
I20260806 09:10:41.879832 6497 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a571ae98049cfa9e2f57d7eae3170 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:10:41 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:55276
I20260806 09:10:41.880614 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6009
I20260806 09:10:41.892743 6825 tablet_replica.cc:330] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: stopping tablet replica
I20260806 09:10:41.893232 6825 raft_consensus.cc:2229] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:41.893635 6825 raft_consensus.cc:2258] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:41.894290 6828 tablet_replica.cc:330] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: stopping tablet replica
I20260806 09:10:41.894430 6831 tablet_replica.cc:330] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: stopping tablet replica
I20260806 09:10:41.894631 6828 raft_consensus.cc:2229] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:10:41.894681 6831 raft_consensus.cc:2229] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:10:41.894830 6825 ts_tablet_manager.cc:1905] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:41.894891 6828 raft_consensus.cc:2258] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:41.895123 6831 raft_consensus.cc:2258] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:10:41.895882 6828 ts_tablet_manager.cc:1905] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:41.896229 6831 ts_tablet_manager.cc:1905] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:10:41.901528 6831 ts_tablet_manager.cc:1918] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:10:41.901681 6831 log.cc:1196] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/2a1a571ae98049cfa9e2f57d7eae3170
I20260806 09:10:41.901896 6825 ts_tablet_manager.cc:1918] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:10:41.902038 6825 log.cc:1196] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/2a1a571ae98049cfa9e2f57d7eae3170
I20260806 09:10:41.902189 6831 ts_tablet_manager.cc:1939] T 2a1a571ae98049cfa9e2f57d7eae3170 P f1a73751b43f4ae78cea34ef099dbe88: Deleting consensus metadata
I20260806 09:10:41.902556 6825 ts_tablet_manager.cc:1939] T 2a1a571ae98049cfa9e2f57d7eae3170 P 2bf15e6e8be34145b453ff076325278c: Deleting consensus metadata
I20260806 09:10:41.902575 6828 ts_tablet_manager.cc:1918] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:10:41.902697 6828 log.cc:1196] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/2a1a571ae98049cfa9e2f57d7eae3170
I20260806 09:10:41.903203 6828 ts_tablet_manager.cc:1939] T 2a1a571ae98049cfa9e2f57d7eae3170 P 1127b0b9f69145a39405c83ee9ea50e4: Deleting consensus metadata
I20260806 09:10:41.903404 5891 catalog_manager.cc:4841] TS f1a73751b43f4ae78cea34ef099dbe88 (127.3.69.3:45649): tablet 2a1a571ae98049cfa9e2f57d7eae3170 (table test-table [id=557ebf6b108d4b69bdf70c4f7047132e]) successfully deleted
I20260806 09:10:41.903782 5890 catalog_manager.cc:4841] TS 2bf15e6e8be34145b453ff076325278c (127.3.69.1:34777): tablet 2a1a571ae98049cfa9e2f57d7eae3170 (table test-table [id=557ebf6b108d4b69bdf70c4f7047132e]) successfully deleted
I20260806 09:10:41.904335 5891 catalog_manager.cc:4841] TS 1127b0b9f69145a39405c83ee9ea50e4 (127.3.69.2:45565): tablet 2a1a571ae98049cfa9e2f57d7eae3170 (table test-table [id=557ebf6b108d4b69bdf70c4f7047132e]) successfully deleted
Aug 06 09:10:41 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:41.913148 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:41.881623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.3.69.1:34777)
0806 09:10:41.883557 (+ 1934us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:41.883567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:41.884273 (+ 706us) server_negotiation.cc:407] Connection header received
0806 09:10:41.884625 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:41.884634 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:41.884774 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:41.884878 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:41.887281 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:41.888916 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:41.891710 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:41.892731 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:41.892869 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:41.907214 (+ 14345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:41.907231 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:41.907248 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:41.907283 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:41.910019 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:41.910736 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:41.910748 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:41.910750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:41.910889 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:41.911345 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:41.911352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:41.911359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:41.911870 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:41.912175 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:41.912500 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 09:10:41.912729 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1733,"thread_start_us":1593,"threads_started":1}
W20260806 09:10:41.913831 6200 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41108
W20260806 09:10:41.914458 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6009: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:41.914733 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 6009
I20260806 09:10:41.944669 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6212
Aug 06 09:10:41 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:10:41.959690 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:41.945395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.3.69.2:45565)
0806 09:10:41.945957 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:41.945962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:41.945979 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:10:41.946043 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:41.946047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:41.946106 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:41.946198 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:41.947621 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:41.948465 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:41.949905 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:41.950307 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:41.950406 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:41.954456 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:41.954468 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:41.954485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:41.954525 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:41.957266 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:41.957885 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:41.957891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:41.957894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:41.958029 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:41.958349 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:41.958352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:41.958354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:41.958720 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:41.958940 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:41.959230 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 09:10:41.959392 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":306,"threads_started":1}
W20260806 09:10:41.960417 6387 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45982
W20260806 09:10:41.961155 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:41.961334 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 6212
I20260806 09:10:41.992192 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6398
Aug 06 09:10:41 dist-test-slave-w4v5 krb5kdc[5849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:10:42.008602 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:41.993062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.3.69.3:45649)
0806 09:10:41.993766 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:41.993770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:41.993786 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:10:41.993842 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:41.993847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:41.993912 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:41.994029 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:41.995440 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:41.996241 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:41.997539 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:41.997970 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:41.998094 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:42.002557 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:42.002572 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:42.002589 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:42.002631 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:42.005703 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:42.006396 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:42.006405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:42.006407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:42.006551 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:42.006942 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:42.006948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:42.006951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:42.007397 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:42.007743 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:42.008015 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 09:10:42.008255 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":324,"threads_started":1}
W20260806 09:10:42.009284 6577 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55334
W20260806 09:10:42.009938 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6398: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:42.010092 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 6398
I20260806 09:10:42.044202 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 5866
I20260806 09:10:42.045445 5969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:42.161773 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 5866
2026-08-06T09:10:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9281 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:43 dist-test-slave-w4v5 krb5kdc[6840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:43 dist-test-slave-w4v5 krb5kdc[6840](info): set up 2 sockets
Aug 06 09:10:43 dist-test-slave-w4v5 krb5kdc[6840](info): commencing operation
krb5kdc: starting...
W20260806 09:10:44.426493 3348 mini_kdc.cc:121] Time spent starting KDC: real 2.210s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:44 dist-test-slave-w4v5 krb5kdc[6840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:44.638727 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40285
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:40285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:44.968879 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:44.969152 6863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:44.969331 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:44.976397 6863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:44.976495 6863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:44.976610 6863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:44.976703 6863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:44.976794 6863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:44.986315 6863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:40285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:44.987658 6863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:44.988926 6863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:45.000977 6870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:45.000932 6874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:45.001000 6872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:45.001236 6863 server_base.cc:1034] running on GCE node
I20260806 09:10:45.002341 6863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:45.003770 6863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:45.005010 6863 hybrid_clock.cc:648] HybridClock initialized: now 1786007445004965 us; error 31 us; skew 500 ppm
I20260806 09:10:45.005347 6863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:45 dist-test-slave-w4v5 krb5kdc[6840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:45.008666 6863 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:45.010164 6863 webserver.cc:460] Webserver started at http://127.3.69.62:36645/ using document root <none> and password file <none>
I20260806 09:10:45.010866 6863 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:45.011020 6863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:45.011379 6863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:45.014410 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "24b32de73ccc4f9484833b809cd92669"
format_stamp: "Formatted at 2026-08-06 09:10:45 on dist-test-slave-w4v5"
server_key: "76db1bcc37632bf842af4551ef889556"
server_key_iv: "9e2ea51dc241f11cc159857a9e59ade4"
server_key_version: "encryptionkey@0"
I20260806 09:10:45.015293 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "24b32de73ccc4f9484833b809cd92669"
format_stamp: "Formatted at 2026-08-06 09:10:45 on dist-test-slave-w4v5"
server_key: "76db1bcc37632bf842af4551ef889556"
server_key_iv: "9e2ea51dc241f11cc159857a9e59ade4"
server_key_version: "encryptionkey@0"
I20260806 09:10:45.020484 6863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:10:45.023872 6883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:45.025148 6863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:10:45.025342 6863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "24b32de73ccc4f9484833b809cd92669"
format_stamp: "Formatted at 2026-08-06 09:10:45 on dist-test-slave-w4v5"
server_key: "76db1bcc37632bf842af4551ef889556"
server_key_iv: "9e2ea51dc241f11cc159857a9e59ade4"
server_key_version: "encryptionkey@0"
I20260806 09:10:45.025555 6863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:45.039276 6863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:45.043331 6863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:45.043622 6863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:45.062951 6863 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:40285
I20260806 09:10:45.062970 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:40285 every 8 connection(s)
I20260806 09:10:45.064394 6863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:45.068255 6961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:45.068547 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 6863
I20260806 09:10:45.068779 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:45.069418 3348 external_mini_cluster.cc:1428] Setting key 5cf131e61d4901d268856f7bc5a2bf7c
I20260806 09:10:45.080706 6961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669: Bootstrap starting.
Aug 06 09:10:45 dist-test-slave-w4v5 krb5kdc[6840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:45.083935 6961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:45.084964 6961 log.cc:824] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:45.088286 6961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669: No bootstrap required, opened a new log
I20260806 09:10:45.092650 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:45.071442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.3.69.62:40285)
0806 09:10:45.072236 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:45.072248 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:45.072289 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:10:45.074497 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:45.074529 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:45.074923 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:45.075402 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:45.076917 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:45.078234 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:45.079598 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:45.080262 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:45.080474 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:45.084715 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:45.084724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:45.084738 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:45.084780 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:45.088489 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:45.089196 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:45.089203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:45.089209 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:45.089375 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:45.089783 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:45.089787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:45.089789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:45.090356 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:45.090727 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:45.091212 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 09:10:45.091641 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":365,"threads_started":1}
I20260806 09:10:45.096810 6961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b32de73ccc4f9484833b809cd92669" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40285 } }
I20260806 09:10:45.097179 6961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:45.097291 6961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b32de73ccc4f9484833b809cd92669, State: Initialized, Role: FOLLOWER
I20260806 09:10:45.097883 6961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b32de73ccc4f9484833b809cd92669" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40285 } }
I20260806 09:10:45.098124 6961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:45.098268 6961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:45.098455 6961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:45.100812 6961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b32de73ccc4f9484833b809cd92669" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40285 } }
I20260806 09:10:45.101321 6961 leader_election.cc:302] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24b32de73ccc4f9484833b809cd92669; no voters:
I20260806 09:10:45.101980 6961 leader_election.cc:288] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:45.102144 6968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:45.103015 6968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [term 1 LEADER]: Becoming Leader. State: Replica: 24b32de73ccc4f9484833b809cd92669, State: Running, Role: LEADER
I20260806 09:10:45.103586 6968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b32de73ccc4f9484833b809cd92669" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40285 } }
I20260806 09:10:45.104214 6961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:45.106374 6969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24b32de73ccc4f9484833b809cd92669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b32de73ccc4f9484833b809cd92669" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40285 } } }
I20260806 09:10:45.106745 6969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:45.107600 6970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b32de73ccc4f9484833b809cd92669. Latest consensus state: current_term: 1 leader_uuid: "24b32de73ccc4f9484833b809cd92669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b32de73ccc4f9484833b809cd92669" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40285 } } }
I20260806 09:10:45.107933 6970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:45.109035 6975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:45.114626 6975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:45.124070 6975 catalog_manager.cc:1347] Generated new cluster ID: 804f7164c45b473594aa28c7bdcd997b
I20260806 09:10:45.124188 6975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:45.142271 6975 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:45.143662 6975 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:45.154796 6975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24b32de73ccc4f9484833b809cd92669: Generated new TSK 0
I20260806 09:10:45.155854 6975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:10:45.212770 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40285
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:45.547705 6997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:45.548009 6997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:45.548190 6997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:45.555120 6997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:45.555258 6997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:45.555445 6997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:10:45.564635 6997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:45.565883 6997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:45.567416 6997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:45.580782 7006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:45.580916 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:45.580910 7007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:45.581948 6997 server_base.cc:1034] running on GCE node
I20260806 09:10:45.582571 6997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:45.583629 6997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:45.584867 6997 hybrid_clock.cc:648] HybridClock initialized: now 1786007445584815 us; error 52 us; skew 500 ppm
I20260806 09:10:45.585181 6997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:45 dist-test-slave-w4v5 krb5kdc[6840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:45.588469 6997 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:10:45.589948 6997 webserver.cc:460] Webserver started at http://127.3.69.1:40603/ using document root <none> and password file <none>
I20260806 09:10:45.590636 6997 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:45.590786 6997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:45.591184 6997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:45.594048 6997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "6845dd3870944fae97f84614f5b4b6e0"
format_stamp: "Formatted at 2026-08-06 09:10:45 on dist-test-slave-w4v5"
server_key: "975d118224829ca9ecbbb5f5798caab3"
server_key_iv: "dc9a7ddf9083d64748cad31cfe490a14"
server_key_version: "encryptionkey@0"
I20260806 09:10:45.594841 6997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "6845dd3870944fae97f84614f5b4b6e0"
format_stamp: "Formatted at 2026-08-06 09:10:45 on dist-test-slave-w4v5"
server_key: "975d118224829ca9ecbbb5f5798caab3"
server_key_iv: "dc9a7ddf9083d64748cad31cfe490a14"
server_key_version: "encryptionkey@0"
I20260806 09:10:45.599911 6997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:10:45.603250 7026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:45.604774 6997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:10:45.604954 6997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "6845dd3870944fae97f84614f5b4b6e0"
format_stamp: "Formatted at 2026-08-06 09:10:45 on dist-test-slave-w4v5"
server_key: "975d118224829ca9ecbbb5f5798caab3"
server_key_iv: "dc9a7ddf9083d64748cad31cfe490a14"
server_key_version: "encryptionkey@0"
I20260806 09:10:45.605173 6997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:45.630226 6997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:45.634696 6997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:45.635011 6997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:45.636029 6997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:45.637405 6997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:45.637512 6997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:45.637676 6997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:45.637749 6997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:45.671841 6997 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:40443
I20260806 09:10:45.671860 7180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:40443 every 8 connection(s)
I20260806 09:10:45.673151 6997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:10:45.674986 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 6997
I20260806 09:10:45.675271 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:10:45.676100 3348 external_mini_cluster.cc:1428] Setting key bd773ba80ea8b683c6919fdf53a68099
Aug 06 09:10:45 dist-test-slave-w4v5 krb5kdc[6840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:45.693404 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:45.675798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:52693 (local address 127.3.69.62:40285)
0806 09:10:45.676265 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:45.676269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:45.677546 (+ 1277us) server_negotiation.cc:407] Connection header received
0806 09:10:45.679664 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:45.679670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:45.679763 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:45.679889 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:45.681332 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:45.682058 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:45.683663 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:45.684070 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:45.684225 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:45.687794 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:45.687799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:45.687802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:45.687832 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:45.690304 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:45.691059 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:45.691063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:45.691066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:45.691243 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:45.691686 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:45.691692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:45.691695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:45.691956 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:45.692115 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:45.692810 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 09:10:45.693039 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":257,"threads_started":1}
I20260806 09:10:45.694325 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:45.676493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40285 (local address 127.3.69.1:52693)
0806 09:10:45.677363 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:45.677412 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:45.679403 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:45.680119 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:45.680129 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:45.680618 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:45.681164 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:45.681180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:45.682224 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:45.682228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:45.683500 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:45.683517 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:45.684310 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:45.684319 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:45.684562 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:45.685269 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:45.685291 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:45.687559 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:45.690445 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:45.690454 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:45.690468 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:45.690923 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:45.691358 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:45.691364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:45.691368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:45.691555 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:45.692088 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:45.692096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:45.692416 (+ 320us) client_negotiation.cc:769] Sending connection context
0806 09:10:45.692684 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 09:10:45.693183 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":367,"threads_started":1}
I20260806 09:10:45.696012 7185 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40285
I20260806 09:10:45.696336 7185 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
I20260806 09:10:45.697089 7185 heartbeater.cc:511] Master 127.3.69.62:40285 requested a full tablet report, sending...
I20260806 09:10:45.699298 6905 ts_manager.cc:194] Registered new tserver with Master: 6845dd3870944fae97f84614f5b4b6e0 (127.3.69.1:40443)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
I20260806 09:10:45.701125 6905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:52693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:10:45.725966 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40285
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:46.052831 7191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:46.053130 7191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:46.053346 7191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:46.060549 7191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:46.060658 7191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:46.060866 7191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:10:46.070900 7191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:46.072197 7191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:46.073496 7191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:46.086653 7198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:46.086635 7203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:46.086721 7199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:46.087713 7191 server_base.cc:1034] running on GCE node
I20260806 09:10:46.088348 7191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:46.089283 7191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:46.090523 7191 hybrid_clock.cc:648] HybridClock initialized: now 1786007446090483 us; error 54 us; skew 500 ppm
I20260806 09:10:46.090896 7191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:46 dist-test-slave-w4v5 krb5kdc[6840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:46.094149 7191 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:10:46.095628 7191 webserver.cc:460] Webserver started at http://127.3.69.2:44579/ using document root <none> and password file <none>
I20260806 09:10:46.096324 7191 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:46.096474 7191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:46.096793 7191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:46.099726 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "be256a98bb8c4916928bf5dcade2a366"
format_stamp: "Formatted at 2026-08-06 09:10:46 on dist-test-slave-w4v5"
server_key: "b25386e6d83f51376337d731a566de29"
server_key_iv: "4761d57685241c21855fb62a3ff9499a"
server_key_version: "encryptionkey@0"
I20260806 09:10:46.100551 7191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "be256a98bb8c4916928bf5dcade2a366"
format_stamp: "Formatted at 2026-08-06 09:10:46 on dist-test-slave-w4v5"
server_key: "b25386e6d83f51376337d731a566de29"
server_key_iv: "4761d57685241c21855fb62a3ff9499a"
server_key_version: "encryptionkey@0"
I20260806 09:10:46.105988 7191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:10:46.109500 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:46.110841 7191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:10:46.111027 7191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "be256a98bb8c4916928bf5dcade2a366"
format_stamp: "Formatted at 2026-08-06 09:10:46 on dist-test-slave-w4v5"
server_key: "b25386e6d83f51376337d731a566de29"
server_key_iv: "4761d57685241c21855fb62a3ff9499a"
server_key_version: "encryptionkey@0"
I20260806 09:10:46.111335 7191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:46.123149 7191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:46.126765 7191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:46.127000 7191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:46.127943 7191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:46.129220 7191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:46.129288 7191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:46.129412 7191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:46.129468 7191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:46.162551 7191 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:46243
I20260806 09:10:46.162596 7362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:46243 every 8 connection(s)
I20260806 09:10:46.163852 7191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:10:46.164925 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 7191
I20260806 09:10:46.165120 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:10:46.165656 3348 external_mini_cluster.cc:1428] Setting key 9879acccf2157b1d491dfd1b8f4cf403
Aug 06 09:10:46 dist-test-slave-w4v5 krb5kdc[6840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:46.187063 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:46.166756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:41129 (local address 127.3.69.62:40285)
0806 09:10:46.166955 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:46.166961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:46.168377 (+ 1416us) server_negotiation.cc:407] Connection header received
0806 09:10:46.171450 (+ 3073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:46.171454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:46.171527 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:46.171620 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:46.173809 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:46.174517 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:46.176676 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:46.177431 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:46.177551 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:46.181005 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:46.181010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:46.181013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:46.181045 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:46.183770 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:46.184674 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:46.184684 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:46.184686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:46.184817 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:46.185282 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:46.185294 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:46.185296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:46.185559 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:46.185731 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:46.186457 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 09:10:46.186710 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
I20260806 09:10:46.188146 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:46.167336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40285 (local address 127.3.69.2:41129)
0806 09:10:46.168163 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:46.168229 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:46.171124 (+ 2895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:46.171885 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:46.171899 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:46.172764 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:46.173602 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:46.173630 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:46.174678 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:46.174708 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:46.176514 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:46.176531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:46.177630 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:46.177635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:46.177826 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:46.178528 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:46.178550 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:46.180766 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:46.183935 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:46.183945 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:46.183964 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:46.184532 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:46.184937 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:46.184942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:46.184946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:46.185147 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:46.185693 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:46.185701 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:46.186065 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 09:10:46.186339 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 09:10:46.186889 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":341,"threads_started":1}
I20260806 09:10:46.189666 7363 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40285
I20260806 09:10:46.190038 7363 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
I20260806 09:10:46.190938 7363 heartbeater.cc:511] Master 127.3.69.62:40285 requested a full tablet report, sending...
I20260806 09:10:46.192484 6905 ts_manager.cc:194] Registered new tserver with Master: be256a98bb8c4916928bf5dcade2a366 (127.3.69.2:46243)
I20260806 09:10:46.193303 6905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:41129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:10:46.218897 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40285
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:46.549463 7369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:46.549755 7369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:46.549928 7369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:46.557289 7369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:46.557426 7369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:46.557624 7369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:10:46.567258 7369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:46.568653 7369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:46.570004 7369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:46.583765 7378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:46.583822 7374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:46.583878 7376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:46.584451 7369 server_base.cc:1034] running on GCE node
I20260806 09:10:46.585189 7369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:46.586126 7369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:46.587412 7369 hybrid_clock.cc:648] HybridClock initialized: now 1786007446587362 us; error 56 us; skew 500 ppm
I20260806 09:10:46.587759 7369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:46 dist-test-slave-w4v5 krb5kdc[6840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:46.591187 7369 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:10:46.592737 7369 webserver.cc:460] Webserver started at http://127.3.69.3:35601/ using document root <none> and password file <none>
I20260806 09:10:46.593425 7369 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:46.593576 7369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:46.593907 7369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:46.596911 7369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "3656cd91ee51426c9d44833e10561322"
format_stamp: "Formatted at 2026-08-06 09:10:46 on dist-test-slave-w4v5"
server_key: "1a6ab95bbaa8c11521f8a20884156a5c"
server_key_iv: "69b1d322180c28d3a95b7440740931ad"
server_key_version: "encryptionkey@0"
I20260806 09:10:46.597719 7369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "3656cd91ee51426c9d44833e10561322"
format_stamp: "Formatted at 2026-08-06 09:10:46 on dist-test-slave-w4v5"
server_key: "1a6ab95bbaa8c11521f8a20884156a5c"
server_key_iv: "69b1d322180c28d3a95b7440740931ad"
server_key_version: "encryptionkey@0"
I20260806 09:10:46.602813 7369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:10:46.606362 7389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:46.607764 7369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:10:46.607945 7369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "3656cd91ee51426c9d44833e10561322"
format_stamp: "Formatted at 2026-08-06 09:10:46 on dist-test-slave-w4v5"
server_key: "1a6ab95bbaa8c11521f8a20884156a5c"
server_key_iv: "69b1d322180c28d3a95b7440740931ad"
server_key_version: "encryptionkey@0"
I20260806 09:10:46.608165 7369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:46.624935 7369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:46.628818 7369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:46.629155 7369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:46.630155 7369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:46.631589 7369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:46.631695 7369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:46.631850 7369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:46.631923 7369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:46.666246 7369 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:43493
I20260806 09:10:46.666311 7535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:43493 every 8 connection(s)
I20260806 09:10:46.667608 7369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:10:46.669710 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 7369
I20260806 09:10:46.669911 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:10:46.670403 3348 external_mini_cluster.cc:1428] Setting key 304093719082eb3f0bd28822ae3f4076
Aug 06 09:10:46 dist-test-slave-w4v5 krb5kdc[6840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:46.688943 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:46.670441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:49027 (local address 127.3.69.62:40285)
0806 09:10:46.670672 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:46.670679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:46.672303 (+ 1624us) server_negotiation.cc:407] Connection header received
0806 09:10:46.674429 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:46.674435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:46.674536 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:46.674654 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:46.676176 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:46.676994 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:46.678490 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:46.678900 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:46.679043 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:46.682485 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:46.682490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:46.682493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:46.682526 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:46.685730 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:46.686772 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:46.686776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:46.686778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:46.686899 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:46.687399 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:46.687402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:46.687404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:46.687702 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:46.687815 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:46.688492 (+ 677us) server_negotiation.cc:299] Negotiation successful
0806 09:10:46.688687 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 09:10:46.689945 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:46.671047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40285 (local address 127.3.69.3:49027)
0806 09:10:46.672132 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:46.672183 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:46.674175 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:46.674878 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:46.674886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:46.675480 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:46.676003 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:46.676029 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:46.677161 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:46.677165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:46.678349 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:46.678360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:46.679161 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:46.679168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:46.679391 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:46.680058 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:46.680086 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:46.682265 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:46.685894 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:46.685905 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:46.685926 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:46.686641 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:46.687020 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:46.687023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:46.687026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:46.687285 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:46.687812 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:46.687817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:46.688129 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 09:10:46.688369 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 09:10:46.688868 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":505,"threads_started":1}
I20260806 09:10:46.691336 7536 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40285
I20260806 09:10:46.691644 7536 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:46.692435 7536 heartbeater.cc:511] Master 127.3.69.62:40285 requested a full tablet report, sending...
I20260806 09:10:46.693919 6905 ts_manager.cc:194] Registered new tserver with Master: 3656cd91ee51426c9d44833e10561322 (127.3.69.3:43493)
I20260806 09:10:46.694842 6905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:49027
I20260806 09:10:46.698999 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:10:46.704152 7185 heartbeater.cc:503] Master 127.3.69.62:40285 was elected leader, sending a full tablet report...
W20260806 09:10:46.734578 7187 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:10:46.732123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.3.69.62:40285)
0806 09:10:46.732310 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:46.732314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:46.732766 (+ 452us) server_negotiation.cc:407] Connection header received
0806 09:10:46.732921 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:46.732925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:46.732994 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:46.733091 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:46.734383 (+ 1292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35668: BlockingRecv error: recv got EOF from 127.0.0.1:35668 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:10:46.737797 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6997
W20260806 09:10:46.740639 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6997: Not authorized: Client connection negotiation failed: client connection to 127.3.69.1:40443: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:10:46.740834 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 6997
I20260806 09:10:46.765466 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 7191
W20260806 09:10:46.768323 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 7191: Not authorized: Client connection negotiation failed: client connection to 127.3.69.2:46243: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:10:46.768512 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 7191
I20260806 09:10:46.789922 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 7369
W20260806 09:10:46.793269 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 7369: Not authorized: Client connection negotiation failed: client connection to 127.3.69.3:43493: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:10:46.793435 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 7369
I20260806 09:10:46.820112 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 6863
I20260806 09:10:46.821414 6954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:46.929136 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 6863
2026-08-06T09:10:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4764 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:47 dist-test-slave-w4v5 krb5kdc[7564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:47 dist-test-slave-w4v5 krb5kdc[7564](info): set up 2 sockets
Aug 06 09:10:47 dist-test-slave-w4v5 krb5kdc[7564](info): commencing operation
krb5kdc: starting...
W20260806 09:10:48.215524 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:48 dist-test-slave-w4v5 krb5kdc[7564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:48.430881 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:38853
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:38853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:48.759797 7586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:48.760221 7586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:48.760426 7586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:48.769727 7586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:48.769833 7586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:48.769879 7586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:48.769937 7586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:48.769984 7586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:48.779901 7586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:38853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:38853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:48.781103 7586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:48.782337 7586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:48.793785 7596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:48.793870 7593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:48.793839 7594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:48.794999 7586 server_base.cc:1034] running on GCE node
I20260806 09:10:48.795791 7586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:48.797113 7586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:48.798343 7586 hybrid_clock.cc:648] HybridClock initialized: now 1786007448798292 us; error 34 us; skew 500 ppm
I20260806 09:10:48.798749 7586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:48 dist-test-slave-w4v5 krb5kdc[7564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007448, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:48.802022 7586 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:48.803546 7586 webserver.cc:460] Webserver started at http://127.3.69.62:36471/ using document root <none> and password file <none>
I20260806 09:10:48.804211 7586 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:48.804347 7586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:48.804675 7586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:48.807716 7586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "3270a803853f4874967714bddea51700"
format_stamp: "Formatted at 2026-08-06 09:10:48 on dist-test-slave-w4v5"
server_key: "9fddb181982df627fd6a95367b01e9c4"
server_key_iv: "29aa12fb255fbb6f1193a07b805ae0e7"
server_key_version: "encryptionkey@0"
I20260806 09:10:48.808511 7586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "3270a803853f4874967714bddea51700"
format_stamp: "Formatted at 2026-08-06 09:10:48 on dist-test-slave-w4v5"
server_key: "9fddb181982df627fd6a95367b01e9c4"
server_key_iv: "29aa12fb255fbb6f1193a07b805ae0e7"
server_key_version: "encryptionkey@0"
I20260806 09:10:48.813706 7586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:10:48.817418 7608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:48.818928 7586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:10:48.819185 7586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "3270a803853f4874967714bddea51700"
format_stamp: "Formatted at 2026-08-06 09:10:48 on dist-test-slave-w4v5"
server_key: "9fddb181982df627fd6a95367b01e9c4"
server_key_iv: "29aa12fb255fbb6f1193a07b805ae0e7"
server_key_version: "encryptionkey@0"
I20260806 09:10:48.819383 7586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:48.834692 7586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:48.839308 7586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:48.839607 7586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:48.858713 7586 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:38853
I20260806 09:10:48.858753 7678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:38853 every 8 connection(s)
I20260806 09:10:48.860155 7586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:48.861620 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 7586
I20260806 09:10:48.861836 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:48.862689 3348 external_mini_cluster.cc:1428] Setting key b5f79babb207dc0dd740bf1c512bc3ee
I20260806 09:10:48.864979 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:10:48 dist-test-slave-w4v5 krb5kdc[7564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:48.878340 7679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700: Bootstrap starting.
I20260806 09:10:48.881763 7679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:48.882798 7679 log.cc:824] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:48.885712 7679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700: No bootstrap required, opened a new log
I20260806 09:10:48.886049 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:48.864672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.3.69.62:38853)
0806 09:10:48.865480 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:48.865491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:48.865531 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:10:48.867203 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:48.867225 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:48.867595 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:48.867913 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:48.869818 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:48.871267 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:48.872981 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:48.873628 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:48.873758 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:48.878603 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:48.878616 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:48.878630 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:48.878665 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:48.881943 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:48.882638 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:48.882656 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:48.882663 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:48.882841 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:48.883295 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:48.883301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:48.883306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:48.883849 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:48.884151 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:48.884627 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 09:10:48.885035 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":394,"threads_started":1}
I20260806 09:10:48.890048 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:10:48.894078 7679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3270a803853f4874967714bddea51700" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38853 } }
I20260806 09:10:48.894454 7679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:48.894569 7679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3270a803853f4874967714bddea51700, State: Initialized, Role: FOLLOWER
I20260806 09:10:48.895282 7679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3270a803853f4874967714bddea51700" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38853 } }
I20260806 09:10:48.895533 7679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:48.895694 7679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:48.895877 7679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:48.898141 7679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3270a803853f4874967714bddea51700" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38853 } }
I20260806 09:10:48.898734 7679 leader_election.cc:302] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3270a803853f4874967714bddea51700; no voters:
I20260806 09:10:48.899480 7679 leader_election.cc:288] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:48.899577 7687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:48.900967 7687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [term 1 LEADER]: Becoming Leader. State: Replica: 3270a803853f4874967714bddea51700, State: Running, Role: LEADER
I20260806 09:10:48.901539 7687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3270a803853f4874967714bddea51700" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38853 } }
I20260806 09:10:48.901929 7679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:48.904652 7689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3270a803853f4874967714bddea51700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3270a803853f4874967714bddea51700" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38853 } } }
I20260806 09:10:48.905160 7689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:48.904948 7691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3270a803853f4874967714bddea51700. Latest consensus state: current_term: 1 leader_uuid: "3270a803853f4874967714bddea51700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3270a803853f4874967714bddea51700" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38853 } } }
I20260806 09:10:48.905594 7691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:48.910457 7700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:48.916069 7700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:48.925845 7700 catalog_manager.cc:1347] Generated new cluster ID: f0d2cab43ef44db3bd4bf756067371a3
I20260806 09:10:48.926011 7700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:48.952088 7700 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:48.953989 7700 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:48.954900 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:48.945133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.3.69.62:38853)
0806 09:10:48.945337 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:48.945344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:48.945963 (+ 619us) server_negotiation.cc:407] Connection header received
0806 09:10:48.946184 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:48.946190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:48.946272 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:48.946381 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:48.947702 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:48.948974 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:48.950891 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:48.951485 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:48.951661 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:48.952858 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:48.952865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:48.952870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:10:48.953877 (+ 1007us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:48.953957 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:10:48.953998 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:48.954113 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:48.954337 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 09:10:48.954471 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":82}
I20260806 09:10:48.964466 7700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3270a803853f4874967714bddea51700: Generated new TSK 0
I20260806 09:10:48.965528 7700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:10:48.974090 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 7586
I20260806 09:10:48.975419 7674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:49.064713 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 7586
2026-08-06T09:10:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2127 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:49 dist-test-slave-w4v5 krb5kdc[7727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:49 dist-test-slave-w4v5 krb5kdc[7727](info): set up 2 sockets
Aug 06 09:10:49 dist-test-slave-w4v5 krb5kdc[7727](info): commencing operation
krb5kdc: starting...
W20260806 09:10:50.346737 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:50 dist-test-slave-w4v5 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:50.562824 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39055
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:39055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:50.891342 7749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:50.891685 7749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:50.891856 7749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:50.898885 7749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:50.899031 7749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:50.899159 7749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:50.899266 7749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:50.899355 7749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:50.908906 7749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:39055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:50.910238 7749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:50.911580 7749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:50.923588 7756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:50.923661 7757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:50.923630 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:50.924396 7749 server_base.cc:1034] running on GCE node
I20260806 09:10:50.925457 7749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:50.927145 7749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:50.928443 7749 hybrid_clock.cc:648] HybridClock initialized: now 1786007450928317 us; error 48 us; skew 500 ppm
I20260806 09:10:50.928898 7749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:50 dist-test-slave-w4v5 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:50.932148 7749 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:50.933540 7749 webserver.cc:460] Webserver started at http://127.3.69.62:39629/ using document root <none> and password file <none>
I20260806 09:10:50.934180 7749 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:50.934281 7749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:50.934531 7749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:50.937475 7749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "e4ed537825394a4d9c297ce437591f86"
format_stamp: "Formatted at 2026-08-06 09:10:50 on dist-test-slave-w4v5"
server_key: "0feaaaa663fbac2295a355577f7a1e67"
server_key_iv: "b318ca63567dfba87bc322b8a7571422"
server_key_version: "encryptionkey@0"
I20260806 09:10:50.938194 7749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "e4ed537825394a4d9c297ce437591f86"
format_stamp: "Formatted at 2026-08-06 09:10:50 on dist-test-slave-w4v5"
server_key: "0feaaaa663fbac2295a355577f7a1e67"
server_key_iv: "b318ca63567dfba87bc322b8a7571422"
server_key_version: "encryptionkey@0"
I20260806 09:10:50.943270 7749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 09:10:50.946429 7770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:50.947784 7749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:10:50.947969 7749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "e4ed537825394a4d9c297ce437591f86"
format_stamp: "Formatted at 2026-08-06 09:10:50 on dist-test-slave-w4v5"
server_key: "0feaaaa663fbac2295a355577f7a1e67"
server_key_iv: "b318ca63567dfba87bc322b8a7571422"
server_key_version: "encryptionkey@0"
I20260806 09:10:50.948159 7749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:50.972009 7749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:50.976051 7749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:50.976352 7749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:50.995571 7749 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:39055
I20260806 09:10:50.995633 7833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:39055 every 8 connection(s)
I20260806 09:10:50.996932 7749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:51.000603 7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:51.003443 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 7749
I20260806 09:10:51.003703 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:51.004334 3348 external_mini_cluster.cc:1428] Setting key 25c0808c49d18608bf897f7d5550344d
I20260806 09:10:51.012954 7834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86: Bootstrap starting.
I20260806 09:10:51.016901 7834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:51.018280 7834 log.cc:824] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:10:51 dist-test-slave-w4v5 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:51.021842 7834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86: No bootstrap required, opened a new log
I20260806 09:10:51.029049 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:51.006312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.3.69.62:39055)
0806 09:10:51.007143 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:51.007155 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:51.007196 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:10:51.008793 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:51.008816 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:51.009206 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:51.009526 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:51.011380 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:51.012728 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:51.014448 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:51.015192 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:51.015334 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:51.020352 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:51.020362 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:51.020383 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:51.020436 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:51.024438 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:51.025264 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:51.025273 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:51.025281 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:51.025485 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:51.025920 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:51.025925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:51.025929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:51.026587 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:51.026891 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:51.027231 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 09:10:51.027708 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":378,"threads_started":1}
I20260806 09:10:51.032310 7834 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ed537825394a4d9c297ce437591f86" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39055 } }
I20260806 09:10:51.032613 7834 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:51.032691 7834 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ed537825394a4d9c297ce437591f86, State: Initialized, Role: FOLLOWER
I20260806 09:10:51.033223 7834 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ed537825394a4d9c297ce437591f86" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39055 } }
I20260806 09:10:51.033419 7834 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:51.033514 7834 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:51.033640 7834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:51.035972 7834 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ed537825394a4d9c297ce437591f86" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39055 } }
I20260806 09:10:51.036412 7834 leader_election.cc:302] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4ed537825394a4d9c297ce437591f86; no voters:
I20260806 09:10:51.036940 7834 leader_election.cc:288] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:51.037091 7839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:51.037827 7839 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [term 1 LEADER]: Becoming Leader. State: Replica: e4ed537825394a4d9c297ce437591f86, State: Running, Role: LEADER
I20260806 09:10:51.038409 7839 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ed537825394a4d9c297ce437591f86" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39055 } }
I20260806 09:10:51.038969 7834 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:51.041749 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4ed537825394a4d9c297ce437591f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ed537825394a4d9c297ce437591f86" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39055 } } }
I20260806 09:10:51.042050 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:51.045596 7841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4ed537825394a4d9c297ce437591f86. Latest consensus state: current_term: 1 leader_uuid: "e4ed537825394a4d9c297ce437591f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4ed537825394a4d9c297ce437591f86" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39055 } } }
I20260806 09:10:51.046073 7841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:51.047284 7846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:51.052897 7846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:51.060938 7846 catalog_manager.cc:1347] Generated new cluster ID: afa97a8e4e734100b13a7e9e1d2aff88
I20260806 09:10:51.061067 7846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:51.073478 7846 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:51.074693 7846 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:51.083384 7846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4ed537825394a4d9c297ce437591f86: Generated new TSK 0
I20260806 09:10:51.084209 7846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:10:51.135360 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39055
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:51.467612 7866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:51.467872 7866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:51.468065 7866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:51.475234 7866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:51.475348 7866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:51.475526 7866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:10:51.485097 7866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:51.486340 7866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:51.487797 7866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:51.500627 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:51.500725 7876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:51.500685 7879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:51.501645 7866 server_base.cc:1034] running on GCE node
I20260806 09:10:51.502416 7866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:51.503557 7866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:51.504791 7866 hybrid_clock.cc:648] HybridClock initialized: now 1786007451504738 us; error 53 us; skew 500 ppm
I20260806 09:10:51.505103 7866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:51 dist-test-slave-w4v5 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007451, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:51.508379 7866 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:10:51.509886 7866 webserver.cc:460] Webserver started at http://127.3.69.1:44641/ using document root <none> and password file <none>
I20260806 09:10:51.510586 7866 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:51.510712 7866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:51.511037 7866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:51.513990 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "1052d825958047ebbd02c75e30f40f86"
format_stamp: "Formatted at 2026-08-06 09:10:51 on dist-test-slave-w4v5"
server_key: "cbd81171f05215cc254ec5eb319915c6"
server_key_iv: "7e4a99f689034c81a111f939274db8fa"
server_key_version: "encryptionkey@0"
I20260806 09:10:51.514772 7866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "1052d825958047ebbd02c75e30f40f86"
format_stamp: "Formatted at 2026-08-06 09:10:51 on dist-test-slave-w4v5"
server_key: "cbd81171f05215cc254ec5eb319915c6"
server_key_iv: "7e4a99f689034c81a111f939274db8fa"
server_key_version: "encryptionkey@0"
I20260806 09:10:51.520059 7866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 09:10:51.523480 7886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:51.524863 7866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:10:51.525059 7866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "1052d825958047ebbd02c75e30f40f86"
format_stamp: "Formatted at 2026-08-06 09:10:51 on dist-test-slave-w4v5"
server_key: "cbd81171f05215cc254ec5eb319915c6"
server_key_iv: "7e4a99f689034c81a111f939274db8fa"
server_key_version: "encryptionkey@0"
I20260806 09:10:51.525270 7866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:51.538432 7866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:51.542343 7866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:51.542639 7866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:51.543637 7866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:51.544935 7866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:51.545040 7866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:51.545189 7866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:51.545261 7866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:51.579919 7866 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:44101
I20260806 09:10:51.579957 8034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:44101 every 8 connection(s)
I20260806 09:10:51.581247 7866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:10:51.586615 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 7866
I20260806 09:10:51.586841 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:10:51.587599 3348 external_mini_cluster.cc:1428] Setting key e1f23b5bda783fe60f64efc11bb33fec
Aug 06 09:10:51 dist-test-slave-w4v5 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007451, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:51.603210 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:51.584052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:47741 (local address 127.3.69.62:39055)
0806 09:10:51.584496 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:51.584501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:51.585594 (+ 1093us) server_negotiation.cc:407] Connection header received
0806 09:10:51.587555 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:51.587563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:51.587676 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:51.587812 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:51.589723 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:51.590787 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:51.592585 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:51.593310 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:51.593460 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:51.597195 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:51.597204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:51.597208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:51.597248 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:51.599869 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:51.600819 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:51.600823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:51.600825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:51.600954 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:51.601469 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:51.601472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:51.601474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:51.601748 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:51.601861 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:51.602730 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 09:10:51.602891 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260806 09:10:51.604755 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:51.584615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39055 (local address 127.3.69.1:47741)
0806 09:10:51.585392 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:51.585450 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:51.587279 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:51.588010 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:51.588018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:51.588727 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:51.589518 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:51.589545 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:51.591136 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:51.591143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:51.592392 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:51.592410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:51.593545 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:51.593550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:51.593767 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:51.594459 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:51.594481 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:51.596957 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:51.600010 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:51.600022 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:51.600042 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:51.600688 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:51.601075 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:51.601082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:51.601088 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:51.601344 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:51.601877 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:51.601889 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:51.602340 (+ 451us) client_negotiation.cc:769] Sending connection context
0806 09:10:51.602643 (+ 303us) client_negotiation.cc:240] Negotiation successful
0806 09:10:51.603306 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":265,"threads_started":1}
I20260806 09:10:51.606410 8035 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39055
I20260806 09:10:51.606751 8035 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:51.607631 8035 heartbeater.cc:511] Master 127.3.69.62:39055 requested a full tablet report, sending...
I20260806 09:10:51.609715 7789 ts_manager.cc:194] Registered new tserver with Master: 1052d825958047ebbd02c75e30f40f86 (127.3.69.1:44101)
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
I20260806 09:10:51.611258 7789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:47741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:10:51.638619 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39055
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:51.967432 8042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:51.967774 8042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:51.967950 8042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:51.975051 8042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:51.975271 8042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:51.975466 8042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:10:51.985203 8042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:51.986471 8042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:51.987872 8042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:52.002363 8055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:52.002305 8053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:52.002852 8052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:52.003955 8042 server_base.cc:1034] running on GCE node
I20260806 09:10:52.004617 8042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:52.005684 8042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:52.006945 8042 hybrid_clock.cc:648] HybridClock initialized: now 1786007452006886 us; error 47 us; skew 500 ppm
I20260806 09:10:52.007333 8042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:52.011268 8042 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:10:52.012863 8042 webserver.cc:460] Webserver started at http://127.3.69.2:33245/ using document root <none> and password file <none>
I20260806 09:10:52.013599 8042 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:52.013751 8042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:52.014077 8042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:52.017027 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "b3a324f531a443e79a83fa1b83013003"
format_stamp: "Formatted at 2026-08-06 09:10:52 on dist-test-slave-w4v5"
server_key: "6a06a985c38f88fa0bdbb575ad0db0c1"
server_key_iv: "5a9ce00e5668688c5cdb8002a80647e5"
server_key_version: "encryptionkey@0"
I20260806 09:10:52.017839 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "b3a324f531a443e79a83fa1b83013003"
format_stamp: "Formatted at 2026-08-06 09:10:52 on dist-test-slave-w4v5"
server_key: "6a06a985c38f88fa0bdbb575ad0db0c1"
server_key_iv: "5a9ce00e5668688c5cdb8002a80647e5"
server_key_version: "encryptionkey@0"
I20260806 09:10:52.023130 8042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 09:10:52.026608 8064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:52.028250 8042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:10:52.028438 8042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "b3a324f531a443e79a83fa1b83013003"
format_stamp: "Formatted at 2026-08-06 09:10:52 on dist-test-slave-w4v5"
server_key: "6a06a985c38f88fa0bdbb575ad0db0c1"
server_key_iv: "5a9ce00e5668688c5cdb8002a80647e5"
server_key_version: "encryptionkey@0"
I20260806 09:10:52.028673 8042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:52.055526 8042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:52.060042 8042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:52.060384 8042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:52.061409 8042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:52.062793 8042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:52.062901 8042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:52.063051 8042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:52.063148 8042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 09:10:52.097034 8042 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:36893
I20260806 09:10:52.097091 8219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:36893 every 8 connection(s)
I20260806 09:10:52.098313 8042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:10:52.100198 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 8042
I20260806 09:10:52.100384 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:10:52.100988 3348 external_mini_cluster.cc:1428] Setting key 402c83afe9a5a2d021f19f5f87279aeb
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:52.120644 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:52.101212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:39315 (local address 127.3.69.62:39055)
0806 09:10:52.101396 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:52.101400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:52.102948 (+ 1548us) server_negotiation.cc:407] Connection header received
0806 09:10:52.104888 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:52.104896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:52.104988 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:52.105105 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:52.106839 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.108012 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.110168 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.110802 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.110941 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:52.114644 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:52.114649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:52.114653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:52.114685 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:52.117255 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.117970 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.117977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.117985 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.118118 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.118626 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.118634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.118641 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.119007 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:52.119223 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:52.119856 (+ 633us) server_negotiation.cc:299] Negotiation successful
0806 09:10:52.120161 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 09:10:52.121383 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:52.101806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39055 (local address 127.3.69.2:39315)
0806 09:10:52.102780 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:52.102829 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:52.104620 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:52.105364 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:52.105374 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:52.106021 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:52.106648 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:52.106678 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.108163 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.108170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:52.110009 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:52.110025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.111036 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.111043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:52.111300 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:52.112042 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:52.112064 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:52.114419 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:52.117395 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:52.117404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:52.117418 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:52.117839 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:52.118240 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:52.118245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:52.118248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:52.118472 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:52.119174 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:52.119184 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:52.119467 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 09:10:52.119708 (+ 241us) client_negotiation.cc:240] Negotiation successful
0806 09:10:52.120266 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":378,"threads_started":1}
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
I20260806 09:10:52.122977 8220 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39055
I20260806 09:10:52.123337 8220 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:52.124213 8220 heartbeater.cc:511] Master 127.3.69.62:39055 requested a full tablet report, sending...
I20260806 09:10:52.125775 7789 ts_manager.cc:194] Registered new tserver with Master: b3a324f531a443e79a83fa1b83013003 (127.3.69.2:36893)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
I20260806 09:10:52.126914 7789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:39315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:10:52.154186 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39055
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:52.484867 8225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:52.485176 8225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:52.485354 8225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:52.492403 8225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:52.492558 8225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:52.492745 8225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:10:52.502219 8225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:52.503559 8225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:52.504912 8225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:52.518471 8234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:52.518626 8238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:52.518659 8231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:52.519348 8225 server_base.cc:1034] running on GCE node
I20260806 09:10:52.519942 8225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:52.520855 8225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:52.522097 8225 hybrid_clock.cc:648] HybridClock initialized: now 1786007452522045 us; error 50 us; skew 500 ppm
I20260806 09:10:52.522421 8225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:52.525609 8225 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:10:52.526991 8225 webserver.cc:460] Webserver started at http://127.3.69.3:43767/ using document root <none> and password file <none>
I20260806 09:10:52.527722 8225 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:52.527872 8225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:52.528189 8225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:52.531172 8225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "e20f39038c5a456fb95254019db30b3b"
format_stamp: "Formatted at 2026-08-06 09:10:52 on dist-test-slave-w4v5"
server_key: "3993b9f63b675acad9393475708d1395"
server_key_iv: "bd4c5bdbbae1db02df502f219e90f7d4"
server_key_version: "encryptionkey@0"
I20260806 09:10:52.532059 8225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "e20f39038c5a456fb95254019db30b3b"
format_stamp: "Formatted at 2026-08-06 09:10:52 on dist-test-slave-w4v5"
server_key: "3993b9f63b675acad9393475708d1395"
server_key_iv: "bd4c5bdbbae1db02df502f219e90f7d4"
server_key_version: "encryptionkey@0"
I20260806 09:10:52.537182 8225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:10:52.540755 8249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:52.542142 8225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 09:10:52.542318 8225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "e20f39038c5a456fb95254019db30b3b"
format_stamp: "Formatted at 2026-08-06 09:10:52 on dist-test-slave-w4v5"
server_key: "3993b9f63b675acad9393475708d1395"
server_key_iv: "bd4c5bdbbae1db02df502f219e90f7d4"
server_key_version: "encryptionkey@0"
I20260806 09:10:52.542515 8225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:52.589622 8225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:52.594161 8225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:52.594517 8225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:52.595701 8225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:52.597213 8225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:52.597330 8225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:52.597533 8225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:52.597620 8225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:52.614217 8035 heartbeater.cc:503] Master 127.3.69.62:39055 was elected leader, sending a full tablet report...
I20260806 09:10:52.633020 8225 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:37295
I20260806 09:10:52.633105 8406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:37295 every 8 connection(s)
I20260806 09:10:52.634311 8225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:10:52.635749 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 8225
I20260806 09:10:52.635915 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:10:52.636478 3348 external_mini_cluster.cc:1428] Setting key 13b993dc114d70e0f3131e5f5aa739bf
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:52.654624 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:52.637189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:32953 (local address 127.3.69.62:39055)
0806 09:10:52.637669 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:52.637674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:52.639167 (+ 1493us) server_negotiation.cc:407] Connection header received
0806 09:10:52.640549 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:52.640554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:52.640642 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:52.640759 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:52.642202 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.642998 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.644535 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.644946 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.645086 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:52.648641 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:52.648647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:52.648650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:52.648687 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:52.651546 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.652379 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.652383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.652385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.652518 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.652956 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.652960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.652962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.653212 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:52.653368 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:52.654022 (+ 654us) server_negotiation.cc:299] Negotiation successful
0806 09:10:52.654239 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":252,"threads_started":1}
I20260806 09:10:52.655551 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:52.637610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39055 (local address 127.3.69.3:32953)
0806 09:10:52.638410 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:52.638455 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:52.640273 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:52.640998 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:52.641007 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:52.641485 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:52.642032 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:52.642049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.643181 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.643186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:52.644382 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:52.644394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.645173 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.645181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:52.645405 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:52.646122 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:52.646143 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:52.648435 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:52.651690 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:52.651700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:52.651714 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:52.652245 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:52.652631 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:52.652636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:52.652639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:52.652826 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:52.653341 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:52.653348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:52.653671 (+ 323us) client_negotiation.cc:769] Sending connection context
0806 09:10:52.653924 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 09:10:52.654417 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":354,"threads_started":1}
I20260806 09:10:52.656843 8407 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39055
I20260806 09:10:52.657106 8407 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:52.657763 8407 heartbeater.cc:511] Master 127.3.69.62:39055 requested a full tablet report, sending...
I20260806 09:10:52.659211 7789 ts_manager.cc:194] Registered new tserver with Master: e20f39038c5a456fb95254019db30b3b (127.3.69.3:37295)
I20260806 09:10:52.659996 7789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:32953
I20260806 09:10:52.664429 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:52.715734 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:52.702042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.3.69.62:39055)
0806 09:10:52.702319 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:52.702326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:52.702631 (+ 305us) server_negotiation.cc:407] Connection header received
0806 09:10:52.702785 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:52.702789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:52.702874 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:52.702963 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:52.704393 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.705175 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.706363 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.706767 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.706858 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:52.710651 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:52.710655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:52.710659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:52.710692 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:52.713420 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.714001 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.714007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.714010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.714123 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.714451 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.714454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.714455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.714790 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:52.714964 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:52.715235 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 09:10:52.715391 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 09:10:52.716300 7789 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59894
I20260806 09:10:52.718055 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 7866
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:52.733438 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:52.719109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.3.69.1:44101)
0806 09:10:52.719776 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:52.719784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:52.719808 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:10:52.719865 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:52.719871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:52.720076 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:52.720191 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:52.721297 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.722271 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.723547 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.724119 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.724227 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:52.727893 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:52.727904 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:52.727917 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:52.727954 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:52.730840 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.731512 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.731519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.731523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.731648 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.731998 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.732002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.732004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.732360 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:52.732653 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:52.732830 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 09:10:52.733035 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":338,"threads_started":1}
W20260806 09:10:52.734737 8030 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36248
W20260806 09:10:52.735903 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 7866: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:52.736129 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 7866
I20260806 09:10:52.764297 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 8042
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:10:52.780434 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:52.765300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.3.69.2:36893)
0806 09:10:52.765812 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:52.765820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:52.765846 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:10:52.765908 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:52.765914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:52.766131 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:52.766306 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:52.767680 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.768746 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.770030 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.770587 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.770743 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:52.775164 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:52.775175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:52.775189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:52.775228 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:52.777821 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.778446 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.778453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.778455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.778573 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.778905 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.778908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.778910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.779306 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:52.779549 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:52.779839 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 09:10:52.780082 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
W20260806 09:10:52.781740 8214 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34484
W20260806 09:10:52.782828 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 8042: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:52.782992 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 8042
I20260806 09:10:52.810283 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 8225
Aug 06 09:10:52 dist-test-slave-w4v5 krb5kdc[7727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007452, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:10:52.827270 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:52.811407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.3.69.3:37295)
0806 09:10:52.812100 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:52.812106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:52.812137 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:10:52.812202 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:52.812209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:52.812425 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:52.812559 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:52.813928 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.815612 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.816836 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:52.817378 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:52.817480 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:52.821750 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:52.821761 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:52.821776 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:52.821810 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:52.824563 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.825217 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.825224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.825226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.825346 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:52.825651 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:52.825654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:52.825656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:52.826000 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:52.826240 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:52.826486 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 09:10:52.826703 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":360,"threads_started":1}
W20260806 09:10:52.828639 8400 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42422
W20260806 09:10:52.829794 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 8225: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:10:52.829948 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 8225
I20260806 09:10:52.855232 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 7749
I20260806 09:10:52.856479 7828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:52.959626 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 7749
2026-08-06T09:10:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3899 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:53 dist-test-slave-w4v5 krb5kdc[8435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:53 dist-test-slave-w4v5 krb5kdc[8435](info): set up 2 sockets
Aug 06 09:10:53 dist-test-slave-w4v5 krb5kdc[8435](info): commencing operation
krb5kdc: starting...
W20260806 09:10:54.249964 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:54 dist-test-slave-w4v5 krb5kdc[8435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:54.482479 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:34071
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:34071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:54.811305 8451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:54.811604 8451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:54.811776 8451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:54.819205 8451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:54.819325 8451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:54.819463 8451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:54.819581 8451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:54.819675 8451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:54.829873 8451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:34071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:34071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:54.831339 8451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:54.832703 8451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:54.845294 8467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:54.845337 8462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:54.845382 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:54.848798 8451 server_base.cc:1034] running on GCE node
I20260806 09:10:54.849663 8451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:54.851224 8451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:54.852437 8451 hybrid_clock.cc:648] HybridClock initialized: now 1786007454852395 us; error 42 us; skew 500 ppm
I20260806 09:10:54.852810 8451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:54 dist-test-slave-w4v5 krb5kdc[8435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:54.856014 8451 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:54.857595 8451 webserver.cc:460] Webserver started at http://127.3.69.62:39787/ using document root <none> and password file <none>
I20260806 09:10:54.858309 8451 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:54.858458 8451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:54.858784 8451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:54.861788 8451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "a3f3e2822f9b4f479df5015fd8db3dbf"
format_stamp: "Formatted at 2026-08-06 09:10:54 on dist-test-slave-w4v5"
server_key: "e7a8f6c312cca3317bf2ea2f43902cf7"
server_key_iv: "840f5d1cc51fd5218b316a90fce7c708"
server_key_version: "encryptionkey@0"
I20260806 09:10:54.862568 8451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "a3f3e2822f9b4f479df5015fd8db3dbf"
format_stamp: "Formatted at 2026-08-06 09:10:54 on dist-test-slave-w4v5"
server_key: "e7a8f6c312cca3317bf2ea2f43902cf7"
server_key_iv: "840f5d1cc51fd5218b316a90fce7c708"
server_key_version: "encryptionkey@0"
I20260806 09:10:54.867645 8451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:10:54.871063 8474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:54.872360 8451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:10:54.872550 8451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "a3f3e2822f9b4f479df5015fd8db3dbf"
format_stamp: "Formatted at 2026-08-06 09:10:54 on dist-test-slave-w4v5"
server_key: "e7a8f6c312cca3317bf2ea2f43902cf7"
server_key_iv: "840f5d1cc51fd5218b316a90fce7c708"
server_key_version: "encryptionkey@0"
I20260806 09:10:54.872745 8451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:54.885042 8451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:54.888801 8451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:54.889074 8451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:54.908195 8451 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:34071
I20260806 09:10:54.908241 8546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:34071 every 8 connection(s)
I20260806 09:10:54.909554 8451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:54.911840 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 8451
I20260806 09:10:54.912070 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:54.912724 3348 external_mini_cluster.cc:1428] Setting key cd82dce938e6891b51d8c00569ba06dd
I20260806 09:10:54.913497 8547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:10:54 dist-test-slave-w4v5 krb5kdc[8435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:54.928377 8547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf: Bootstrap starting.
I20260806 09:10:54.932633 8547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:54.934054 8547 log.cc:824] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:54.936575 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:54.914621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.3.69.62:34071)
0806 09:10:54.915404 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:54.915415 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:54.915455 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:10:54.916984 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:54.917005 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:54.917372 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:54.917688 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:54.919571 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:54.920917 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:54.922649 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:54.923366 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:54.923489 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:54.928155 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:54.928166 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:54.928185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:54.928229 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:54.932180 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:54.932920 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:54.932929 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:54.932938 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:54.933217 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:54.933607 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:54.933615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:54.933621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:54.934367 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:54.934700 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:54.935211 (+ 511us) server_negotiation.cc:299] Negotiation successful
0806 09:10:54.935600 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":431,"threads_started":1}
I20260806 09:10:54.937126 8547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf: No bootstrap required, opened a new log
I20260806 09:10:54.944541 8547 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f3e2822f9b4f479df5015fd8db3dbf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34071 } }
I20260806 09:10:54.944900 8547 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:54.945032 8547 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f3e2822f9b4f479df5015fd8db3dbf, State: Initialized, Role: FOLLOWER
I20260806 09:10:54.945775 8547 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f3e2822f9b4f479df5015fd8db3dbf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34071 } }
I20260806 09:10:54.946062 8547 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:54.946200 8547 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:54.946333 8547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:54.948362 8547 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f3e2822f9b4f479df5015fd8db3dbf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34071 } }
I20260806 09:10:54.948766 8547 leader_election.cc:302] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3f3e2822f9b4f479df5015fd8db3dbf; no voters:
I20260806 09:10:54.949333 8547 leader_election.cc:288] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:54.949486 8552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:54.950881 8552 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [term 1 LEADER]: Becoming Leader. State: Replica: a3f3e2822f9b4f479df5015fd8db3dbf, State: Running, Role: LEADER
I20260806 09:10:54.951494 8547 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:54.951395 8552 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f3e2822f9b4f479df5015fd8db3dbf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34071 } }
I20260806 09:10:54.956163 8557 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3f3e2822f9b4f479df5015fd8db3dbf. Latest consensus state: current_term: 1 leader_uuid: "a3f3e2822f9b4f479df5015fd8db3dbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f3e2822f9b4f479df5015fd8db3dbf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34071 } } }
I20260806 09:10:54.956914 8557 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:54.956394 8554 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3f3e2822f9b4f479df5015fd8db3dbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f3e2822f9b4f479df5015fd8db3dbf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 34071 } } }
I20260806 09:10:54.957127 8554 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:54.958527 8564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:54.964222 8564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:54.973203 8564 catalog_manager.cc:1347] Generated new cluster ID: fd54912890954a249eb8c58e30284e24
I20260806 09:10:54.973325 8564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:54.990886 8564 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:54.992746 8564 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:55.016369 8564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3f3e2822f9b4f479df5015fd8db3dbf: Generated new TSK 0
I20260806 09:10:55.017493 8564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:10:55.082472 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34071
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:55.420132 8585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:55.420413 8585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:55.420579 8585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:55.427887 8585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:55.428026 8585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:55.428215 8585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:10:55.438053 8585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:55.439383 8585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:55.440661 8585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:55.453837 8600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:55.453876 8597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:55.453918 8595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:55.454803 8585 server_base.cc:1034] running on GCE node
I20260806 09:10:55.455452 8585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:55.456358 8585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:55.457588 8585 hybrid_clock.cc:648] HybridClock initialized: now 1786007455457529 us; error 55 us; skew 500 ppm
I20260806 09:10:55.457944 8585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:55 dist-test-slave-w4v5 krb5kdc[8435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:55.461193 8585 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:10:55.462494 8585 webserver.cc:460] Webserver started at http://127.3.69.1:41451/ using document root <none> and password file <none>
I20260806 09:10:55.463189 8585 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:55.463299 8585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:55.463568 8585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:55.467433 8585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "6b1ac894357042c3b64b4203dde8b6b6"
format_stamp: "Formatted at 2026-08-06 09:10:55 on dist-test-slave-w4v5"
server_key: "4cd594fb38311182372c65d382b241f8"
server_key_iv: "091f4d8206764a8bc310545f667c09ce"
server_key_version: "encryptionkey@0"
I20260806 09:10:55.468372 8585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "6b1ac894357042c3b64b4203dde8b6b6"
format_stamp: "Formatted at 2026-08-06 09:10:55 on dist-test-slave-w4v5"
server_key: "4cd594fb38311182372c65d382b241f8"
server_key_iv: "091f4d8206764a8bc310545f667c09ce"
server_key_version: "encryptionkey@0"
I20260806 09:10:55.473441 8585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 09:10:55.476874 8609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:55.478101 8585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:10:55.478281 8585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "6b1ac894357042c3b64b4203dde8b6b6"
format_stamp: "Formatted at 2026-08-06 09:10:55 on dist-test-slave-w4v5"
server_key: "4cd594fb38311182372c65d382b241f8"
server_key_iv: "091f4d8206764a8bc310545f667c09ce"
server_key_version: "encryptionkey@0"
I20260806 09:10:55.478497 8585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:55.502972 8585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:55.506973 8585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:55.507301 8585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:55.508208 8585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:55.509457 8585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:55.509531 8585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:55.509624 8585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:55.509660 8585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:55.545126 8585 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:39129
I20260806 09:10:55.545235 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:39129 every 8 connection(s)
I20260806 09:10:55.546461 8585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:10:55.555532 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 8585
I20260806 09:10:55.555805 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:10:55.556459 3348 external_mini_cluster.cc:1428] Setting key 66ffbed1121b3ba81d064ff9a8986bd2
Aug 06 09:10:55 dist-test-slave-w4v5 krb5kdc[8435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:55.566929 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:55.549127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:36167 (local address 127.3.69.62:34071)
0806 09:10:55.549558 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:55.549562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:55.550546 (+ 984us) server_negotiation.cc:407] Connection header received
0806 09:10:55.552674 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:55.552681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:55.552785 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:55.552912 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:55.554404 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:55.555203 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:55.556890 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:55.557309 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:55.557454 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:55.561145 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:55.561151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:55.561156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:55.561196 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:55.563881 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:55.564619 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:55.564626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:55.564634 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:55.564755 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:55.565192 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:55.565198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:55.565201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:55.565467 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:55.565641 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:55.566327 (+ 686us) server_negotiation.cc:299] Negotiation successful
0806 09:10:55.566547 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":212,"threads_started":1}
I20260806 09:10:55.567901 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:55.549655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34071 (local address 127.3.69.1:36167)
0806 09:10:55.550365 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:55.550403 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:55.552373 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:55.553161 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:55.553170 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:55.553668 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:55.554222 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:55.554239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:55.555371 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:55.555376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:55.556746 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:55.556759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:55.557541 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:55.557551 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:55.557843 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:55.558620 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:55.558642 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:55.560938 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:55.564028 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:55.564037 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:55.564053 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:55.564486 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:55.564872 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:55.564876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:55.564880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:55.565069 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:55.565611 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:55.565619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:55.565945 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 09:10:55.566216 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 09:10:55.566730 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":278,"threads_started":1}
I20260806 09:10:55.569512 8765 heartbeater.cc:347] Connected to a master server at 127.3.69.62:34071
I20260806 09:10:55.569845 8765 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:55.570767 8765 heartbeater.cc:511] Master 127.3.69.62:34071 requested a full tablet report, sending...
I20260806 09:10:55.573187 8498 ts_manager.cc:194] Registered new tserver with Master: 6b1ac894357042c3b64b4203dde8b6b6 (127.3.69.1:39129)
I20260806 09:10:55.575397 8498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:36167
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:10:55.608155 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34071
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:55.934378 8774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:55.934710 8774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:55.934897 8774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:55.942163 8774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:55.942305 8774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:55.942487 8774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:10:55.952270 8774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:55.953622 8774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:55.954958 8774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:55.968468 8782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:55.968508 8780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:55.968748 8785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:55.969615 8774 server_base.cc:1034] running on GCE node
I20260806 09:10:55.970217 8774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:55.971248 8774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:55.972458 8774 hybrid_clock.cc:648] HybridClock initialized: now 1786007455972409 us; error 45 us; skew 500 ppm
I20260806 09:10:55.972831 8774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:55 dist-test-slave-w4v5 krb5kdc[8435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:55.976666 8774 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:10:55.978269 8774 webserver.cc:460] Webserver started at http://127.3.69.2:44385/ using document root <none> and password file <none>
I20260806 09:10:55.978914 8774 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:55.979017 8774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:55.979346 8774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:55.982343 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "90319a8d2e7f4141808cf633b90196d8"
format_stamp: "Formatted at 2026-08-06 09:10:55 on dist-test-slave-w4v5"
server_key: "a45fabd96a581cc69a76ce1a3113eb22"
server_key_iv: "0ebaa27b059dbfa26391b3377440bd3e"
server_key_version: "encryptionkey@0"
I20260806 09:10:55.983196 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "90319a8d2e7f4141808cf633b90196d8"
format_stamp: "Formatted at 2026-08-06 09:10:55 on dist-test-slave-w4v5"
server_key: "a45fabd96a581cc69a76ce1a3113eb22"
server_key_iv: "0ebaa27b059dbfa26391b3377440bd3e"
server_key_version: "encryptionkey@0"
I20260806 09:10:55.988397 8774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:10:55.991891 8795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:55.993271 8774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 09:10:55.993414 8774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "90319a8d2e7f4141808cf633b90196d8"
format_stamp: "Formatted at 2026-08-06 09:10:55 on dist-test-slave-w4v5"
server_key: "a45fabd96a581cc69a76ce1a3113eb22"
server_key_iv: "0ebaa27b059dbfa26391b3377440bd3e"
server_key_version: "encryptionkey@0"
I20260806 09:10:55.993597 8774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:56.009680 8774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:56.014643 8774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:56.014969 8774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:56.015976 8774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:56.017621 8774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:56.017741 8774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:56.017897 8774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:56.017971 8774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:56.050614 8774 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:36273
I20260806 09:10:56.050657 8937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:36273 every 8 connection(s)
I20260806 09:10:56.052080 8774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:10:56.058152 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 8774
I20260806 09:10:56.058393 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:10:56.058977 3348 external_mini_cluster.cc:1428] Setting key 8e7581f3407236ecb05ce4301b39c108
Aug 06 09:10:56 dist-test-slave-w4v5 krb5kdc[8435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:56.072970 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.054706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:48629 (local address 127.3.69.62:34071)
0806 09:10:56.054886 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:56.054890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:56.056069 (+ 1179us) server_negotiation.cc:407] Connection header received
0806 09:10:56.057954 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:56.057958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:56.058033 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:56.058138 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:56.060159 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.061094 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.062715 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.063277 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.063425 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.066924 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:56.066929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:56.066933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:56.066963 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:56.069825 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.070591 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.070598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.070601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.070745 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.071238 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.071244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.071248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.071514 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:56.071674 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:56.072361 (+ 687us) server_negotiation.cc:299] Negotiation successful
0806 09:10:56.072598 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:10:56.073956 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.055203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34071 (local address 127.3.69.2:48629)
0806 09:10:56.055908 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:56.055945 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:56.057614 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:56.058431 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:56.058445 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:56.059200 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:56.059962 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:56.059986 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.061272 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.061277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:56.062558 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:56.062575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.063515 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.063522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:56.063772 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.064471 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:56.064493 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:56.066695 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:56.069978 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:56.069989 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:56.070005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:56.070438 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:56.070866 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:56.070871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:56.070875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:56.071116 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:56.071659 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:56.071666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:56.072002 (+ 336us) client_negotiation.cc:769] Sending connection context
0806 09:10:56.072276 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 09:10:56.072819 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":277,"threads_started":1}
I20260806 09:10:56.075608 8939 heartbeater.cc:347] Connected to a master server at 127.3.69.62:34071
I20260806 09:10:56.075935 8939 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:56.076766 8939 heartbeater.cc:511] Master 127.3.69.62:34071 requested a full tablet report, sending...
I20260806 09:10:56.078346 8498 ts_manager.cc:194] Registered new tserver with Master: 90319a8d2e7f4141808cf633b90196d8 (127.3.69.2:36273)
I20260806 09:10:56.079236 8498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:48629
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:10:56.111301 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34071
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:56.442924 8946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:56.443285 8946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:56.443463 8946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:56.450615 8946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:56.450716 8946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:56.450917 8946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:10:56.460954 8946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:34071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:56.462280 8946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:56.463760 8946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:56.477572 8953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:56.477555 8954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:56.477937 8956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:56.478503 8946 server_base.cc:1034] running on GCE node
I20260806 09:10:56.479066 8946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:56.480161 8946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:56.481365 8946 hybrid_clock.cc:648] HybridClock initialized: now 1786007456481309 us; error 46 us; skew 500 ppm
I20260806 09:10:56.481695 8946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:56 dist-test-slave-w4v5 krb5kdc[8435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:56.484905 8946 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:10:56.486295 8946 webserver.cc:460] Webserver started at http://127.3.69.3:40041/ using document root <none> and password file <none>
I20260806 09:10:56.486979 8946 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:56.487151 8946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:56.487480 8946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:56.490383 8946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "b7cf65d82bb941e08d799764b585d83f"
format_stamp: "Formatted at 2026-08-06 09:10:56 on dist-test-slave-w4v5"
server_key: "aae8fb5df0f2cc9f550a721b7896833d"
server_key_iv: "2f9a7e757776651f82987551c4b1878b"
server_key_version: "encryptionkey@0"
I20260806 09:10:56.491304 8946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "b7cf65d82bb941e08d799764b585d83f"
format_stamp: "Formatted at 2026-08-06 09:10:56 on dist-test-slave-w4v5"
server_key: "aae8fb5df0f2cc9f550a721b7896833d"
server_key_iv: "2f9a7e757776651f82987551c4b1878b"
server_key_version: "encryptionkey@0"
I20260806 09:10:56.496860 8946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:10:56.500487 8967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:56.501787 8946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 09:10:56.501981 8946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "b7cf65d82bb941e08d799764b585d83f"
format_stamp: "Formatted at 2026-08-06 09:10:56 on dist-test-slave-w4v5"
server_key: "aae8fb5df0f2cc9f550a721b7896833d"
server_key_iv: "2f9a7e757776651f82987551c4b1878b"
server_key_version: "encryptionkey@0"
I20260806 09:10:56.502198 8946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:56.520617 8946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:56.524682 8946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:56.524981 8946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:56.525949 8946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:56.527354 8946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:56.527462 8946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:56.527619 8946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:56.527700 8946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:56.562582 8946 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:34863
I20260806 09:10:56.562671 9115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:34863 every 8 connection(s)
I20260806 09:10:56.563970 8946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:10:56.572391 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 8946
I20260806 09:10:56.572654 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:10:56.573097 3348 external_mini_cluster.cc:1428] Setting key 80c2d177dad8e6b57f20583152bca917
Aug 06 09:10:56 dist-test-slave-w4v5 krb5kdc[8435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:56.579227 8765 heartbeater.cc:503] Master 127.3.69.62:34071 was elected leader, sending a full tablet report...
I20260806 09:10:56.585182 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.566621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:54749 (local address 127.3.69.62:34071)
0806 09:10:56.566808 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:56.566812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:56.567977 (+ 1165us) server_negotiation.cc:407] Connection header received
0806 09:10:56.569815 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:56.569820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:56.569913 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:56.570028 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:56.571606 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.572381 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.574194 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.574878 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.575074 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.578839 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:56.578845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:56.578850 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:56.578942 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:56.582019 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.582779 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.582785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.582789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.582925 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.583412 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.583419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.583427 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.583688 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:56.583860 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:56.584577 (+ 717us) server_negotiation.cc:299] Negotiation successful
0806 09:10:56.584822 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:10:56.586099 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.567052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:34071 (local address 127.3.69.3:54749)
0806 09:10:56.567807 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:56.567857 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:56.569547 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:56.570265 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:56.570273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:56.570783 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:56.571424 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:56.571443 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.572551 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.572555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:56.573992 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:56.574006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.575244 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.575253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:56.575489 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.576252 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:56.576281 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:56.578622 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:56.582165 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:56.582174 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:56.582192 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:56.582637 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:56.583043 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:56.583047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:56.583051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:56.583284 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:56.583834 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:56.583842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:56.584187 (+ 345us) client_negotiation.cc:769] Sending connection context
0806 09:10:56.584469 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 09:10:56.584994 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":278,"threads_started":1}
I20260806 09:10:56.587798 9116 heartbeater.cc:347] Connected to a master server at 127.3.69.62:34071
I20260806 09:10:56.588147 9116 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:56.589061 9116 heartbeater.cc:511] Master 127.3.69.62:34071 requested a full tablet report, sending...
I20260806 09:10:56.590499 8498 ts_manager.cc:194] Registered new tserver with Master: b7cf65d82bb941e08d799764b585d83f (127.3.69.3:34863)
I20260806 09:10:56.591361 8498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:54749
I20260806 09:10:56.601478 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:10:56 dist-test-slave-w4v5 krb5kdc[8435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:10:56 dist-test-slave-w4v5 krb5kdc[8435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:10:56.655805 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.640835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.3.69.1:39129)
0806 09:10:56.641454 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:56.641461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:56.641488 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:10:56.641552 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:56.641558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:56.641773 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:56.641921 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:56.643470 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.644501 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.645701 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.646267 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.646364 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.650300 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:56.650311 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:56.650326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:56.650362 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:56.653252 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.653885 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.653892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.653894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.654012 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.654331 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.654334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.654336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.654661 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:56.654888 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:56.655201 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 09:10:56.655405 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":300,"threads_started":1}
Aug 06 09:10:56 dist-test-slave-w4v5 krb5kdc[8435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:56.674276 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.660788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.3.69.62:34071)
0806 09:10:56.660988 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:56.660993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:56.661244 (+ 251us) server_negotiation.cc:407] Connection header received
0806 09:10:56.661373 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:56.661377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:56.661443 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:56.661530 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:56.662759 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.663673 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.664893 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.665319 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.665433 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.669310 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:56.669316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:56.669320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:56.669355 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:56.672103 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.672682 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.672685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.672688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.672802 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.673116 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.673120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.673121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.673426 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:56.673618 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:56.673858 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 09:10:56.674042 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 09:10:56.674962 8498 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48630
I20260806 09:10:56.676194 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 8585
I20260806 09:10:56.689177 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.677233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.3.69.1:39129)
0806 09:10:56.677526 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:56.677532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:56.677694 (+ 162us) server_negotiation.cc:407] Connection header received
0806 09:10:56.677849 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:56.677852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:56.677915 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:56.678004 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:56.679219 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.679909 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.681089 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.681645 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.681759 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.684120 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:56.684125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:56.684128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:56.684160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:56.686842 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.687528 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.687533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.687535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.687665 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.687985 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.687989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.687991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.688243 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:56.688427 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:56.688668 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 09:10:56.688828 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 09:10:56.689790 8760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:56.774677 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 8585
I20260806 09:10:56.801335 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 8774
Aug 06 09:10:56 dist-test-slave-w4v5 krb5kdc[8435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:10:56.817577 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.802279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.3.69.2:36273)
0806 09:10:56.802798 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:56.802805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:56.802830 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:10:56.802890 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:56.802896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:56.803188 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:56.803371 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:56.804673 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.805724 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.807316 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.807888 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.807984 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.812209 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:56.812220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:56.812234 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:56.812267 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:56.815008 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.815724 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.815730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.815733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.815853 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.816168 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.816171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.816173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.816500 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:56.816740 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:56.816978 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 09:10:56.817176 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
I20260806 09:10:56.818944 8933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:56.911135 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 8774
I20260806 09:10:56.936841 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 8946
Aug 06 09:10:56 dist-test-slave-w4v5 krb5kdc[8435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:10:56.954140 9133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:56.937891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.3.69.3:34863)
0806 09:10:56.938583 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:56.938591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:56.938622 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:10:56.938712 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:56.938718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:56.938939 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:56.939069 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:56.940384 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.941436 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.942666 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:56.943380 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:56.943479 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:56.947837 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:56.947851 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:56.947868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:56.947916 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:56.951228 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.951936 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.951944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.951946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.952075 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:56.952459 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:56.952464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:56.952467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:56.952834 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:56.953188 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:56.953496 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 09:10:56.953785 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":348,"threads_started":1}
I20260806 09:10:56.955538 9110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:57.038580 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 8946
I20260806 09:10:57.060648 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 8451
I20260806 09:10:57.062079 8542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:10:57.165134 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 8451
2026-08-06T09:10:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4206 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:10:57 dist-test-slave-w4v5 krb5kdc[9140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:10:57 dist-test-slave-w4v5 krb5kdc[9140](info): set up 2 sockets
Aug 06 09:10:57 dist-test-slave-w4v5 krb5kdc[9140](info): commencing operation
krb5kdc: starting...
W20260806 09:10:58.462203 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:10:58 dist-test-slave-w4v5 krb5kdc[9140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:10:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:10:58.687341 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36177
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:59.018781 9158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:59.019124 9158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:59.019261 9158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:59.026356 9158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:10:59.026461 9158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:59.026511 9158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:59.026568 9158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:10:59.026616 9158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:10:59.036491 9158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:36177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:59.037745 9158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:59.038973 9158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:59.051290 9168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:59.051515 9170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:59.051290 9167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:59.052142 9158 server_base.cc:1034] running on GCE node
I20260806 09:10:59.052992 9158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:59.054411 9158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:59.055685 9158 hybrid_clock.cc:648] HybridClock initialized: now 1786007459055645 us; error 54 us; skew 500 ppm
I20260806 09:10:59.056063 9158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:59 dist-test-slave-w4v5 krb5kdc[9140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007459, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:59.059410 9158 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:10:59.060833 9158 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:59.061053 9158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:59.061403 9158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:59.064420 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "1d713a80c8554d568a6cc2e4dbd39122"
format_stamp: "Formatted at 2026-08-06 09:10:59 on dist-test-slave-w4v5"
server_key: "560ea58b24ff6e5c9ac57151b2749d74"
server_key_iv: "fff7b84e0534754e11161c3f3e1e6a2f"
server_key_version: "encryptionkey@0"
I20260806 09:10:59.065277 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "1d713a80c8554d568a6cc2e4dbd39122"
format_stamp: "Formatted at 2026-08-06 09:10:59 on dist-test-slave-w4v5"
server_key: "560ea58b24ff6e5c9ac57151b2749d74"
server_key_iv: "fff7b84e0534754e11161c3f3e1e6a2f"
server_key_version: "encryptionkey@0"
I20260806 09:10:59.070389 9158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:10:59.074452 9176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:59.075973 9158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:10:59.076186 9158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "1d713a80c8554d568a6cc2e4dbd39122"
format_stamp: "Formatted at 2026-08-06 09:10:59 on dist-test-slave-w4v5"
server_key: "560ea58b24ff6e5c9ac57151b2749d74"
server_key_iv: "fff7b84e0534754e11161c3f3e1e6a2f"
server_key_version: "encryptionkey@0"
I20260806 09:10:59.076432 9158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:59.088369 9158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:59.092708 9158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:59.093007 9158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:59.112202 9158 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:36177
I20260806 09:10:59.112226 9248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:36177 every 8 connection(s)
I20260806 09:10:59.113245 9158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:10:59.116971 9249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:10:59.118042 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 9158
I20260806 09:10:59.118296 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:10:59.119228 3348 external_mini_cluster.cc:1428] Setting key 7c248fa10ed54476b0ef5b7b985eb75e
I20260806 09:10:59.131919 9249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122: Bootstrap starting.
Aug 06 09:10:59 dist-test-slave-w4v5 krb5kdc[9140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:59.135625 9249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122: Neither blocks nor log segments found. Creating new log.
I20260806 09:10:59.136673 9249 log.cc:824] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122: Log is configured to *not* fsync() on all Append() calls
I20260806 09:10:59.139603 9249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122: No bootstrap required, opened a new log
I20260806 09:10:59.142527 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:59.121417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.3.69.62:36177)
0806 09:10:59.122272 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:59.122284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:59.122326 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:10:59.123981 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:59.124010 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:59.124383 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:59.124729 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:59.126546 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:59.127910 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:59.129785 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:59.130441 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:59.130594 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:59.135251 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:59.135259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:59.135273 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:59.135308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:59.138466 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:59.139180 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:59.139192 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:59.139199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:59.139366 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:59.139751 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:59.139758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:59.139761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:59.140280 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:59.140603 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:59.141065 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 09:10:59.141518 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":413,"threads_started":1}
I20260806 09:10:59.147583 9249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d713a80c8554d568a6cc2e4dbd39122" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36177 } }
I20260806 09:10:59.147998 9249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:10:59.148111 9249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d713a80c8554d568a6cc2e4dbd39122, State: Initialized, Role: FOLLOWER
I20260806 09:10:59.148649 9249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d713a80c8554d568a6cc2e4dbd39122" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36177 } }
I20260806 09:10:59.148919 9249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:10:59.149075 9249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:10:59.149317 9249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:10:59.151785 9249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d713a80c8554d568a6cc2e4dbd39122" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36177 } }
I20260806 09:10:59.152344 9249 leader_election.cc:302] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d713a80c8554d568a6cc2e4dbd39122; no voters:
I20260806 09:10:59.153065 9249 leader_election.cc:288] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:10:59.153218 9254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:10:59.154024 9254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [term 1 LEADER]: Becoming Leader. State: Replica: 1d713a80c8554d568a6cc2e4dbd39122, State: Running, Role: LEADER
I20260806 09:10:59.154601 9254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d713a80c8554d568a6cc2e4dbd39122" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36177 } }
I20260806 09:10:59.155308 9249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:10:59.157622 9257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d713a80c8554d568a6cc2e4dbd39122. Latest consensus state: current_term: 1 leader_uuid: "1d713a80c8554d568a6cc2e4dbd39122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d713a80c8554d568a6cc2e4dbd39122" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36177 } } }
I20260806 09:10:59.158074 9257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:59.158727 9255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d713a80c8554d568a6cc2e4dbd39122" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d713a80c8554d568a6cc2e4dbd39122" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36177 } } }
I20260806 09:10:59.159129 9255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122 [sys.catalog]: This master's current role is: LEADER
I20260806 09:10:59.159255 9267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:10:59.165819 9267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:10:59.174206 9267 catalog_manager.cc:1347] Generated new cluster ID: 7c0ac09422614bc8bed4557309bb66e3
I20260806 09:10:59.174335 9267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:10:59.188166 9267 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:10:59.189451 9267 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:10:59.198161 9267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122: Generated new TSK 0
I20260806 09:10:59.198982 9267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:10:59.255904 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36177
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:10:59.596371 9285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:10:59.596645 9285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:10:59.596841 9285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:10:59.604151 9285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:10:59.604256 9285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:10:59.604467 9285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:10:59.614096 9285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:10:59.615410 9285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:10:59.616760 9285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:10:59.629585 9292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:59.629546 9293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:10:59.629506 9297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:10:59.630338 9285 server_base.cc:1034] running on GCE node
I20260806 09:10:59.630959 9285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:10:59.631946 9285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:10:59.633177 9285 hybrid_clock.cc:648] HybridClock initialized: now 1786007459633132 us; error 67 us; skew 500 ppm
I20260806 09:10:59.633490 9285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:10:59 dist-test-slave-w4v5 krb5kdc[9140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007459, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:10:59.636767 9285 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:10:59.638016 9285 fs_manager.cc:359] Metadata directory not provided
I20260806 09:10:59.638156 9285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:10:59.638547 9285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:10:59.641402 9285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "63c032dd8d214995b724df46f99e522f"
format_stamp: "Formatted at 2026-08-06 09:10:59 on dist-test-slave-w4v5"
server_key: "782295228c6f826fb7769cee82c55c94"
server_key_iv: "d73001ec08bf4e5d22277fe0e4aaf02d"
server_key_version: "encryptionkey@0"
I20260806 09:10:59.642320 9285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "63c032dd8d214995b724df46f99e522f"
format_stamp: "Formatted at 2026-08-06 09:10:59 on dist-test-slave-w4v5"
server_key: "782295228c6f826fb7769cee82c55c94"
server_key_iv: "d73001ec08bf4e5d22277fe0e4aaf02d"
server_key_version: "encryptionkey@0"
I20260806 09:10:59.647426 9285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260806 09:10:59.650952 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:59.652295 9285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:10:59.652477 9285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "63c032dd8d214995b724df46f99e522f"
format_stamp: "Formatted at 2026-08-06 09:10:59 on dist-test-slave-w4v5"
server_key: "782295228c6f826fb7769cee82c55c94"
server_key_iv: "d73001ec08bf4e5d22277fe0e4aaf02d"
server_key_version: "encryptionkey@0"
I20260806 09:10:59.652714 9285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:10:59.669684 9285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:10:59.674031 9285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:10:59.674317 9285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:10:59.675364 9285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:10:59.676712 9285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:10:59.676842 9285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:59.676970 9285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:10:59.677062 9285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:10:59.712406 9285 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:46185
I20260806 09:10:59.712419 9475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:46185 every 8 connection(s)
I20260806 09:10:59.713296 9285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:10:59.719728 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 9285
I20260806 09:10:59.719949 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:10:59.720597 3348 external_mini_cluster.cc:1428] Setting key 5208bf08a645a8459d5cb6c4a8ef76be
Aug 06 09:10:59 dist-test-slave-w4v5 krb5kdc[9140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007459, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:10:59.733306 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:59.715829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:51937 (local address 127.3.69.62:36177)
0806 09:10:59.716309 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 09:10:59.716314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:10:59.717169 (+ 855us) server_negotiation.cc:407] Connection header received
0806 09:10:59.718966 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:10:59.718973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:10:59.719067 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:10:59.719247 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:10:59.720927 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:59.721670 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:59.723268 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:10:59.723676 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:10:59.723819 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:59.727340 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:10:59.727345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:10:59.727348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:10:59.727380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:10:59.729918 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:59.730668 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:59.730675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:59.730680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:59.730837 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:10:59.731336 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:10:59.731343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:10:59.731349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:10:59.731719 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:10:59.731878 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:10:59.732553 (+ 675us) server_negotiation.cc:299] Negotiation successful
0806 09:10:59.732812 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":269,"threads_started":1}
I20260806 09:10:59.733901 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:10:59.716275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36177 (local address 127.3.69.1:51937)
0806 09:10:59.717006 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 09:10:59.717049 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 09:10:59.718697 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:10:59.719491 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:10:59.719500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:10:59.720140 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:10:59.720742 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:59.720778 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:59.721839 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:59.721844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:59.723064 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:10:59.723130 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:10:59.723909 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:10:59.723918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:10:59.724155 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:10:59.724864 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:10:59.724886 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:10:59.727060 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:10:59.730069 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:59.730077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:59.730089 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:59.730544 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:59.730962 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:10:59.730965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:10:59.730967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:10:59.731223 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:10:59.731843 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:10:59.731849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:10:59.732151 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 09:10:59.732390 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 09:10:59.732770 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":325,"threads_started":1}
I20260806 09:10:59.735541 9477 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36177
I20260806 09:10:59.735895 9477 heartbeater.cc:464] Registering TS with master...
I20260806 09:10:59.736714 9477 heartbeater.cc:511] Master 127.3.69.62:36177 requested a full tablet report, sending...
I20260806 09:10:59.738941 9199 ts_manager.cc:194] Registered new tserver with Master: 63c032dd8d214995b724df46f99e522f (127.3.69.1:46185)
I20260806 09:10:59.740536 9199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:51937
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:10:59.775552 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36177
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:00.110390 9484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:00.110720 9484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:00.110893 9484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:00.122219 9484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:00.122387 9484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:00.122611 9484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:00.135524 9484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:00.136761 9484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:00.138094 9484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:00.150870 9494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:00.150899 9495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:00.151247 9502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:00.152096 9484 server_base.cc:1034] running on GCE node
I20260806 09:11:00.152652 9484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:00.153527 9484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:00.154733 9484 hybrid_clock.cc:648] HybridClock initialized: now 1786007460154697 us; error 32 us; skew 500 ppm
I20260806 09:11:00.155051 9484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:00 dist-test-slave-w4v5 krb5kdc[9140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007460, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:00.158552 9484 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:11:00.159788 9484 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:00.159904 9484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:00.160257 9484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:00.163165 9484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "645ef77a3f214159a14267b406b846b5"
format_stamp: "Formatted at 2026-08-06 09:11:00 on dist-test-slave-w4v5"
server_key: "30f8e1603cd94196f0a1bc5d715a7d96"
server_key_iv: "bd784052de2d5b193b2ea7a38451f3d9"
server_key_version: "encryptionkey@0"
I20260806 09:11:00.164006 9484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "645ef77a3f214159a14267b406b846b5"
format_stamp: "Formatted at 2026-08-06 09:11:00 on dist-test-slave-w4v5"
server_key: "30f8e1603cd94196f0a1bc5d715a7d96"
server_key_iv: "bd784052de2d5b193b2ea7a38451f3d9"
server_key_version: "encryptionkey@0"
I20260806 09:11:00.169322 9484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 09:11:00.172775 9510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:00.174034 9484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:11:00.174222 9484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "645ef77a3f214159a14267b406b846b5"
format_stamp: "Formatted at 2026-08-06 09:11:00 on dist-test-slave-w4v5"
server_key: "30f8e1603cd94196f0a1bc5d715a7d96"
server_key_iv: "bd784052de2d5b193b2ea7a38451f3d9"
server_key_version: "encryptionkey@0"
I20260806 09:11:00.174436 9484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:00.187929 9484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:00.192004 9484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:00.192291 9484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:00.193253 9484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:00.194594 9484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:00.194684 9484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:00.194815 9484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:00.194914 9484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:00.229586 9484 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:46093
I20260806 09:11:00.229681 9665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:46093 every 8 connection(s)
I20260806 09:11:00.230588 9484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:00.239112 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 9484
I20260806 09:11:00.239328 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:11:00.239807 3348 external_mini_cluster.cc:1428] Setting key 1ad2cb4a16f36bbcda8b96775b7057bc
Aug 06 09:11:00 dist-test-slave-w4v5 krb5kdc[9140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007460, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:00.250800 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.233146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:60081 (local address 127.3.69.62:36177)
0806 09:11:00.233338 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.233343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.234592 (+ 1249us) server_negotiation.cc:407] Connection header received
0806 09:11:00.236482 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.236487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.236583 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.236717 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:00.238217 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.238928 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.240727 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.241413 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.241561 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.245022 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:00.245027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:00.245030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:00.245061 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:00.247744 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:00.248481 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:00.248493 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:00.248497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:00.248625 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:00.249078 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:00.249081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:00.249083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:00.249347 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:00.249505 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:00.250193 (+ 688us) server_negotiation.cc:299] Negotiation successful
0806 09:11:00.250417 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:11:00.251827 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.233624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36177 (local address 127.3.69.2:60081)
0806 09:11:00.234337 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:00.234463 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:00.236191 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:00.236945 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:00.236953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:00.237480 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:00.238034 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.238050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.239113 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.239119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.240569 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.240583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.241691 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.241698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.241883 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.242569 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:00.242596 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:00.244793 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:00.247893 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:00.247901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:00.247917 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:00.248347 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:00.248753 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:00.248758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:00.248762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:00.248946 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:00.249480 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:00.249487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:00.249828 (+ 341us) client_negotiation.cc:769] Sending connection context
0806 09:11:00.250090 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 09:11:00.250653 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":292,"threads_started":1}
I20260806 09:11:00.253301 9666 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36177
I20260806 09:11:00.253638 9666 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:00.254462 9666 heartbeater.cc:511] Master 127.3.69.62:36177 requested a full tablet report, sending...
I20260806 09:11:00.256317 9199 ts_manager.cc:194] Registered new tserver with Master: 645ef77a3f214159a14267b406b846b5 (127.3.69.2:46093)
I20260806 09:11:00.257596 9199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:60081
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:11:00.294741 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36177
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:00.627457 9673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:00.627740 9673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:00.627840 9673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:00.635582 9673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:00.635689 9673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:00.635807 9673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:00.645478 9673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:00.646739 9673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:00.648191 9673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:00.661547 9679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:00.661499 9682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:00.661428 9680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:00.662544 9673 server_base.cc:1034] running on GCE node
I20260806 09:11:00.663256 9673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:00.664216 9673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:00.665463 9673 hybrid_clock.cc:648] HybridClock initialized: now 1786007460665435 us; error 50 us; skew 500 ppm
I20260806 09:11:00.665884 9673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:00 dist-test-slave-w4v5 krb5kdc[9140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007460, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:00.669396 9673 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:11:00.670526 9673 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:00.670619 9673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:00.670882 9673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:00.673770 9673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "04b6f27e8db248069e73395f1725889b"
format_stamp: "Formatted at 2026-08-06 09:11:00 on dist-test-slave-w4v5"
server_key: "0d4cb2f91a04b12affe2a9ca9424b769"
server_key_iv: "4cb7fc371a507f7ea6720f917d7a073f"
server_key_version: "encryptionkey@0"
I20260806 09:11:00.674542 9673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "04b6f27e8db248069e73395f1725889b"
format_stamp: "Formatted at 2026-08-06 09:11:00 on dist-test-slave-w4v5"
server_key: "0d4cb2f91a04b12affe2a9ca9424b769"
server_key_iv: "4cb7fc371a507f7ea6720f917d7a073f"
server_key_version: "encryptionkey@0"
I20260806 09:11:00.679668 9673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 09:11:00.683243 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:00.684535 9673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:11:00.684746 9673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "04b6f27e8db248069e73395f1725889b"
format_stamp: "Formatted at 2026-08-06 09:11:00 on dist-test-slave-w4v5"
server_key: "0d4cb2f91a04b12affe2a9ca9424b769"
server_key_iv: "4cb7fc371a507f7ea6720f917d7a073f"
server_key_version: "encryptionkey@0"
I20260806 09:11:00.684967 9673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:00.709647 9673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:00.714826 9673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:00.715165 9673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:00.716408 9673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:00.718014 9673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:00.718129 9673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:00.718242 9673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:00.718319 9673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:00.744017 9477 heartbeater.cc:503] Master 127.3.69.62:36177 was elected leader, sending a full tablet report...
I20260806 09:11:00.751935 9673 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:36505
I20260806 09:11:00.752018 9845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:36505 every 8 connection(s)
I20260806 09:11:00.752876 9673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:00.757319 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 9673
I20260806 09:11:00.757813 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:11:00.758541 3348 external_mini_cluster.cc:1428] Setting key 276698d3302e9b00d5c883e0be0e9d43
Aug 06 09:11:00 dist-test-slave-w4v5 krb5kdc[9140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007460, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:00.775959 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.755407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:46137 (local address 127.3.69.62:36177)
0806 09:11:00.755918 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.755923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.760214 (+ 4291us) server_negotiation.cc:407] Connection header received
0806 09:11:00.760288 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.760297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.760391 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.760518 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:00.762578 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.763755 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.765662 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.766100 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.766243 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.769737 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:00.769743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:00.769748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:00.769794 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:00.772400 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:00.773116 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:00.773121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:00.773125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:00.773253 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:00.773830 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:00.773836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:00.773838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:00.774215 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:00.774380 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:00.775362 (+ 982us) server_negotiation.cc:299] Negotiation successful
0806 09:11:00.775618 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":292,"threads_started":1}
I20260806 09:11:00.777328 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.756001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36177 (local address 127.3.69.3:46137)
0806 09:11:00.756953 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:00.757003 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:00.759828 (+ 2825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:00.760806 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:00.760823 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:00.761627 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:00.762400 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.762419 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.763907 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.763914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.765481 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.765493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.766344 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.766351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.766578 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.767321 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:00.767343 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:00.769501 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:00.772542 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:00.772551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:00.772567 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:00.772987 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:00.773369 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:00.773374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:00.773377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:00.773668 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:00.774398 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:00.774409 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:00.774878 (+ 469us) client_negotiation.cc:769] Sending connection context
0806 09:11:00.775247 (+ 369us) client_negotiation.cc:240] Negotiation successful
0806 09:11:00.775776 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":377,"threads_started":1}
I20260806 09:11:00.778708 9847 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36177
I20260806 09:11:00.779031 9847 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:00.779747 9847 heartbeater.cc:511] Master 127.3.69.62:36177 requested a full tablet report, sending...
I20260806 09:11:00.781275 9199 ts_manager.cc:194] Registered new tserver with Master: 04b6f27e8db248069e73395f1725889b (127.3.69.3:36505)
I20260806 09:11:00.782087 9199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:46137
I20260806 09:11:00.788028 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:11:00.805735 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.793159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.3.69.62:36177)
0806 09:11:00.793346 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.793350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.793746 (+ 396us) server_negotiation.cc:407] Connection header received
0806 09:11:00.793878 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.793882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.793944 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.794030 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:00.795635 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.796372 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.797643 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.798061 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.798157 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.800482 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:00.800487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:00.800490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:00.800522 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:00.803449 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:00.804069 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:00.804073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:00.804076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:00.804205 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:00.804531 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:00.804534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:00.804536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:00.804860 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:00.805039 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:00.805331 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 09:11:00.805500 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:11:00.810249 9199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:11:00.817451 9199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:00.858937 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.831731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.3.69.3:36505)
0806 09:11:00.848784 (+ 17053us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.848798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.849594 (+ 796us) server_negotiation.cc:407] Connection header received
0806 09:11:00.849750 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.849763 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.850106 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.850315 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:00.851470 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.852807 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.855460 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.857632 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.857774 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.858021 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:00.858338 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 09:11:00.858542 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16768,"thread_start_us":17075,"threads_started":1}
I20260806 09:11:00.859122 9873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.831050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:36505 (local address 127.0.0.1:35202)
0806 09:11:00.849233 (+ 18183us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:00.849260 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:00.849432 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:00.850706 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:00.850713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:00.850743 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:00.851208 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.851223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.852946 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.852954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.855254 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.855269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.857839 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.857845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.858046 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.858139 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 09:11:00.858279 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 09:11:00.858490 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16755,"mutex_wait_us":1,"thread_start_us":11361,"threads_started":1}
I20260806 09:11:00.859920 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.832296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.3.69.1:46185)
0806 09:11:00.849877 (+ 17581us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.849890 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.850853 (+ 963us) server_negotiation.cc:407] Connection header received
0806 09:11:00.850960 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.850970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.851211 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.851407 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:00.852199 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.853696 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.856211 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.857824 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.857961 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.858200 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:00.859403 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0806 09:11:00.859586 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17343,"thread_start_us":17609,"threads_started":1}
I20260806 09:11:00.860059 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.831109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.3.69.2:46093)
0806 09:11:00.850534 (+ 19425us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.850544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.850573 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:11:00.850689 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.850697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.850927 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.851728 (+ 801us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:00.851969 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.853510 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.856434 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.858088 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.858212 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.858406 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:00.859684 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0806 09:11:00.859816 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19232,"thread_start_us":19430,"threads_started":1}
I20260806 09:11:00.859267 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.831565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46185 (local address 127.0.0.1:42948)
0806 09:11:00.850324 (+ 18759us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:00.850342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:00.850457 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:00.851518 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:00.851525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:00.851561 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:00.852002 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.852018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.853843 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.853848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.855988 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.856001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.858341 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.858347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.858498 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.858534 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:11:00.858627 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 09:11:00.858792 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18603,"mutex_wait_us":43,"thread_start_us":16035,"threads_started":1}
I20260806 09:11:00.860531 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.830874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:46093 (local address 127.0.0.1:32886)
0806 09:11:00.842897 (+ 12023us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:00.842947 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:00.844017 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:00.851173 (+ 7156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:00.851182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:00.851206 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:00.851735 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.851750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.853670 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.853675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.856222 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.856238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.858603 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.858611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.858862 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.858897 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:11:00.858988 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 09:11:00.859173 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11715,"mutex_wait_us":101,"thread_start_us":19126,"threads_started":1}
I20260806 09:11:00.863374 9374 tablet_service.cc:1467] Processing CreateTablet for tablet 24a45e82eff24cb69d486354293e48c7 (DEFAULT_TABLE table=test-table [id=955407717299472f994eae57733a70f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:00.863444 9751 tablet_service.cc:1467] Processing CreateTablet for tablet 24a45e82eff24cb69d486354293e48c7 (DEFAULT_TABLE table=test-table [id=955407717299472f994eae57733a70f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:00.863914 9574 tablet_service.cc:1467] Processing CreateTablet for tablet 24a45e82eff24cb69d486354293e48c7 (DEFAULT_TABLE table=test-table [id=955407717299472f994eae57733a70f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:00.864833 9374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a45e82eff24cb69d486354293e48c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:00.864910 9751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a45e82eff24cb69d486354293e48c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:00.865191 9574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a45e82eff24cb69d486354293e48c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:00.875869 9876 tablet_bootstrap.cc:492] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Bootstrap starting.
I20260806 09:11:00.879416 9876 tablet_bootstrap.cc:654] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:00.879755 9877 tablet_bootstrap.cc:492] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Bootstrap starting.
I20260806 09:11:00.881178 9876 log.cc:824] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:00.883612 9877 tablet_bootstrap.cc:654] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:00.884687 9876 tablet_bootstrap.cc:492] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: No bootstrap required, opened a new log
I20260806 09:11:00.884864 9877 log.cc:824] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:00.884965 9876 ts_tablet_manager.cc:1397] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260806 09:11:00.886016 9878 tablet_bootstrap.cc:492] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Bootstrap starting.
I20260806 09:11:00.887917 9877 tablet_bootstrap.cc:492] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: No bootstrap required, opened a new log
I20260806 09:11:00.888234 9877 ts_tablet_manager.cc:1397] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260806 09:11:00.889616 9878 tablet_bootstrap.cc:654] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:00.890950 9878 log.cc:824] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:00.894164 9878 tablet_bootstrap.cc:492] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: No bootstrap required, opened a new log
I20260806 09:11:00.894439 9878 ts_tablet_manager.cc:1397] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260806 09:11:00.895957 9876 raft_consensus.cc:348] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
I20260806 09:11:00.896085 9877 raft_consensus.cc:348] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
I20260806 09:11:00.896407 9876 raft_consensus.cc:374] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:00.896433 9877 raft_consensus.cc:374] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:00.896509 9876 raft_consensus.cc:729] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c032dd8d214995b724df46f99e522f, State: Initialized, Role: FOLLOWER
I20260806 09:11:00.896525 9877 raft_consensus.cc:729] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b6f27e8db248069e73395f1725889b, State: Initialized, Role: FOLLOWER
I20260806 09:11:00.897137 9877 consensus_queue.cc:260] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
I20260806 09:11:00.897238 9876 consensus_queue.cc:260] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
I20260806 09:11:00.898231 9847 heartbeater.cc:503] Master 127.3.69.62:36177 was elected leader, sending a full tablet report...
I20260806 09:11:00.898919 9877 ts_tablet_manager.cc:1428] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 09:11:00.905927 9876 ts_tablet_manager.cc:1428] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260806 09:11:00.906545 9878 raft_consensus.cc:348] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
I20260806 09:11:00.906962 9878 raft_consensus.cc:374] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:00.907092 9878 raft_consensus.cc:729] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645ef77a3f214159a14267b406b846b5, State: Initialized, Role: FOLLOWER
I20260806 09:11:00.907732 9878 consensus_queue.cc:260] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
I20260806 09:11:00.908746 9666 heartbeater.cc:503] Master 127.3.69.62:36177 was elected leader, sending a full tablet report...
I20260806 09:11:00.909396 9878 ts_tablet_manager.cc:1428] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260806 09:11:00.945040 9890 raft_consensus.cc:482] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:00.945330 9890 raft_consensus.cc:504] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
I20260806 09:11:00.947125 9890 leader_election.cc:288] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04b6f27e8db248069e73395f1725889b (127.3.69.3:36505), 63c032dd8d214995b724df46f99e522f (127.3.69.1:46185)
I20260806 09:11:00.954106 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.947678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:47277 (local address 127.3.69.1:46185)
0806 09:11:00.947909 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.947918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.948737 (+ 819us) server_negotiation.cc:407] Connection header received
0806 09:11:00.948937 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.948943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.949009 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.949110 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:00.949760 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.950511 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.952404 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.953146 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.953310 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.953426 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:00.953715 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 09:11:00.953836 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:11:00.954183 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.947458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:56019 (local address 127.3.69.3:36505)
0806 09:11:00.947697 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.947704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.948695 (+ 991us) server_negotiation.cc:407] Connection header received
0806 09:11:00.948930 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.948934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.949000 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.949103 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:00.949810 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.950528 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.952416 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.953206 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.953352 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.953468 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:00.953782 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 09:11:00.953918 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:11:00.954277 9893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.947662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46185 (local address 127.3.69.2:47277)
0806 09:11:00.948628 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:00.948647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:00.948802 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:00.949120 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:00.949124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:00.949148 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:00.949572 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.949581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.950645 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.950651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.952252 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.952264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.953397 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.953403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.953545 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.953568 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 09:11:00.953688 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 09:11:00.953816 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":46,"thread_start_us":580,"threads_started":1}
I20260806 09:11:00.954327 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.947558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:36505 (local address 127.3.69.2:56019)
0806 09:11:00.948566 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:00.948583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:00.948759 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:00.949133 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:00.949137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:00.949163 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:00.949655 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.949665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.950668 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.950675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.952250 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:00.952263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:00.953454 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:00.953461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:00.953622 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:00.953645 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 09:11:00.953755 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 09:11:00.953877 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":2,"thread_start_us":515,"threads_started":1}
I20260806 09:11:00.955504 9415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a45e82eff24cb69d486354293e48c7" candidate_uuid: "645ef77a3f214159a14267b406b846b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c032dd8d214995b724df46f99e522f" is_pre_election: true
I20260806 09:11:00.955502 9789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a45e82eff24cb69d486354293e48c7" candidate_uuid: "645ef77a3f214159a14267b406b846b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b6f27e8db248069e73395f1725889b" is_pre_election: true
I20260806 09:11:00.956048 9789 raft_consensus.cc:2454] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645ef77a3f214159a14267b406b846b5 in term 0.
I20260806 09:11:00.956148 9415 raft_consensus.cc:2454] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 645ef77a3f214159a14267b406b846b5 in term 0.
I20260806 09:11:00.956804 9515 leader_election.cc:302] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b6f27e8db248069e73395f1725889b, 645ef77a3f214159a14267b406b846b5; no voters:
I20260806 09:11:00.957338 9890 raft_consensus.cc:2781] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:00.957505 9890 raft_consensus.cc:482] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:00.957613 9890 raft_consensus.cc:3037] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:00.959982 9890 raft_consensus.cc:504] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
I20260806 09:11:00.960669 9890 leader_election.cc:288] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [CANDIDATE]: Term 1 election: Requested vote from peers 04b6f27e8db248069e73395f1725889b (127.3.69.3:36505), 63c032dd8d214995b724df46f99e522f (127.3.69.1:46185)
I20260806 09:11:00.961107 9789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a45e82eff24cb69d486354293e48c7" candidate_uuid: "645ef77a3f214159a14267b406b846b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b6f27e8db248069e73395f1725889b"
I20260806 09:11:00.961110 9415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a45e82eff24cb69d486354293e48c7" candidate_uuid: "645ef77a3f214159a14267b406b846b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c032dd8d214995b724df46f99e522f"
I20260806 09:11:00.961335 9789 raft_consensus.cc:3037] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:00.961331 9415 raft_consensus.cc:3037] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:00.963708 9415 raft_consensus.cc:2454] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645ef77a3f214159a14267b406b846b5 in term 1.
I20260806 09:11:00.963804 9789 raft_consensus.cc:2454] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 645ef77a3f214159a14267b406b846b5 in term 1.
I20260806 09:11:00.964162 9512 leader_election.cc:302] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c032dd8d214995b724df46f99e522f, 645ef77a3f214159a14267b406b846b5; no voters:
I20260806 09:11:00.964504 9890 raft_consensus.cc:2781] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:00.964962 9890 raft_consensus.cc:686] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 1 LEADER]: Becoming Leader. State: Replica: 645ef77a3f214159a14267b406b846b5, State: Running, Role: LEADER
I20260806 09:11:00.965441 9890 consensus_queue.cc:237] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } }
W20260806 09:11:00.965592 9478 tablet.cc:2369] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:00.970288 9198 catalog_manager.cc:5495] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 645ef77a3f214159a14267b406b846b5 (127.3.69.2). New cstate: current_term: 1 leader_uuid: "645ef77a3f214159a14267b406b846b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "645ef77a3f214159a14267b406b846b5" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 46093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 } health_report { overall_health: UNKNOWN } } }
W20260806 09:11:00.982539 9667 tablet.cc:2369] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:01.004338 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:00.998884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.3.69.2:46093)
0806 09:11:00.999127 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:00.999134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:00.999243 (+ 109us) server_negotiation.cc:407] Connection header received
0806 09:11:00.999491 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:00.999495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:00.999572 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:00.999736 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:01.000111 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:01.000815 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:01.002155 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:01.002582 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:01.002723 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:01.003245 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:01.003735 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:01.003899 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:01.003969 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 09:11:01.004054 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 09:11:01.004513 9848 tablet.cc:2369] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:01.012528 9415 raft_consensus.cc:1261] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 1 FOLLOWER]: Refusing update from remote peer 645ef77a3f214159a14267b406b846b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:01.012524 9789 raft_consensus.cc:1261] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 1 FOLLOWER]: Refusing update from remote peer 645ef77a3f214159a14267b406b846b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:01.013542 9895 consensus_queue.cc:1035] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b6f27e8db248069e73395f1725889b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:01.013898 9890 consensus_queue.cc:1035] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c032dd8d214995b724df46f99e522f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:01.024890 9898 mvcc.cc:205] Tried to move back new op lower bound from 7315486560297238528 to 7315486560118542336. Current Snapshot: MvccSnapshot[applied={T|T < 7315486560297238528}]
I20260806 09:11:01.026222 9900 mvcc.cc:205] Tried to move back new op lower bound from 7315486560297238528 to 7315486560118542336. Current Snapshot: MvccSnapshot[applied={T|T < 7315486560297238528}]
I20260806 09:11:01.040343 9198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:01.040673 9198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:01.044430 9198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d713a80c8554d568a6cc2e4dbd39122: Sending DeleteTablet for 3 replicas of tablet 24a45e82eff24cb69d486354293e48c7
I20260806 09:11:01.045540 9574 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a45e82eff24cb69d486354293e48c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:01 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:32886
I20260806 09:11:01.045620 9751 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a45e82eff24cb69d486354293e48c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:01 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:35202
I20260806 09:11:01.045634 9374 tablet_service.cc:1514] Processing DeleteTablet for tablet 24a45e82eff24cb69d486354293e48c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:01 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:42948
I20260806 09:11:01.051923 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 9285
I20260806 09:11:01.057071 9911 tablet_replica.cc:330] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: stopping tablet replica
I20260806 09:11:01.057806 9911 raft_consensus.cc:2229] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:01.058467 9911 raft_consensus.cc:2258] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:01.059414 9912 tablet_replica.cc:330] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: stopping tablet replica
I20260806 09:11:01.059726 9909 tablet_replica.cc:330] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: stopping tablet replica
I20260806 09:11:01.059870 9912 raft_consensus.cc:2229] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:01.060189 9909 raft_consensus.cc:2229] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:11:01.060359 9912 raft_consensus.cc:2258] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:01.060945 9909 raft_consensus.cc:2258] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:01.062755 9911 ts_tablet_manager.cc:1905] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:01.063035 9909 ts_tablet_manager.cc:1905] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:01.065689 9912 ts_tablet_manager.cc:1905] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:01.070116 9911 ts_tablet_manager.cc:1918] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:01.070310 9911 log.cc:1196] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/24a45e82eff24cb69d486354293e48c7
I20260806 09:11:01.070540 9909 ts_tablet_manager.cc:1918] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:01.070749 9909 log.cc:1196] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/24a45e82eff24cb69d486354293e48c7
I20260806 09:11:01.070996 9911 ts_tablet_manager.cc:1939] T 24a45e82eff24cb69d486354293e48c7 P 63c032dd8d214995b724df46f99e522f: Deleting consensus metadata
I20260806 09:11:01.071419 9909 ts_tablet_manager.cc:1939] T 24a45e82eff24cb69d486354293e48c7 P 645ef77a3f214159a14267b406b846b5: Deleting consensus metadata
I20260806 09:11:01.072963 9177 catalog_manager.cc:4841] TS 63c032dd8d214995b724df46f99e522f (127.3.69.1:46185): tablet 24a45e82eff24cb69d486354293e48c7 (table test-table [id=955407717299472f994eae57733a70f0]) successfully deleted
I20260806 09:11:01.072975 9179 catalog_manager.cc:4841] TS 645ef77a3f214159a14267b406b846b5 (127.3.69.2:46093): tablet 24a45e82eff24cb69d486354293e48c7 (table test-table [id=955407717299472f994eae57733a70f0]) successfully deleted
I20260806 09:11:01.076200 9912 ts_tablet_manager.cc:1918] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:01.076377 9912 log.cc:1196] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/24a45e82eff24cb69d486354293e48c7
I20260806 09:11:01.076931 9912 ts_tablet_manager.cc:1939] T 24a45e82eff24cb69d486354293e48c7 P 04b6f27e8db248069e73395f1725889b: Deleting consensus metadata
I20260806 09:11:01.078728 9181 catalog_manager.cc:4841] TS 04b6f27e8db248069e73395f1725889b (127.3.69.3:36505): tablet 24a45e82eff24cb69d486354293e48c7 (table test-table [id=955407717299472f994eae57733a70f0]) successfully deleted
Aug 06 09:11:01 dist-test-slave-w4v5 krb5kdc[9140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:11:01.088090 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:01.053385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.3.69.1:46185)
0806 09:11:01.053566 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:01.053572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:01.067573 (+ 14001us) server_negotiation.cc:407] Connection header received
0806 09:11:01.070671 (+ 3098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:01.070677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:01.070768 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:01.070845 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:01.073071 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:01.074198 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:01.076186 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:01.076917 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:01.077075 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:01.082207 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:01.082219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:01.082233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:01.082271 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:01.085196 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:01.085914 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:01.085921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:01.085923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:01.086066 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:01.086469 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:01.086473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:01.086475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:01.086845 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:01.087219 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:01.087449 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 09:11:01.087758 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:11:01.088922 9469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:01.178839 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 9285
I20260806 09:11:01.204696 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 9484
Aug 06 09:11:01 dist-test-slave-w4v5 krb5kdc[9140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:11:01.223125 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:01.205687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.3.69.2:46093)
0806 09:11:01.205877 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:01.205881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:01.205924 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:11:01.206147 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:01.206151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:01.206222 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:01.206322 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:01.208259 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:01.209126 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:01.210961 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:01.211486 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:01.211634 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:01.216681 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:01.216697 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:01.216714 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:01.216755 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:01.220052 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:01.220746 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:01.220754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:01.220756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:01.220908 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:01.221321 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:01.221326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:01.221330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:01.221735 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:01.222088 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:01.222429 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 09:11:01.222699 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:11:01.223903 9660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:01.337800 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 9484
I20260806 09:11:01.363940 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 9673
Aug 06 09:11:01 dist-test-slave-w4v5 krb5kdc[9140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:11:01.380056 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:01.364713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.3.69.3:36505)
0806 09:11:01.364921 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:01.364928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:01.365035 (+ 107us) server_negotiation.cc:407] Connection header received
0806 09:11:01.365262 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:01.365272 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:01.365341 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:01.365450 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:01.366899 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:01.367696 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:01.369208 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:01.369623 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:01.369761 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:01.374296 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:01.374310 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:01.374328 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:01.374365 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:01.377403 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:01.378073 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:01.378080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:01.378083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:01.378215 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:01.378615 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:01.378618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:01.378620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:01.378974 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:01.379371 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:01.379499 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 09:11:01.379666 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:11:01.380785 9841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:01.470503 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 9673
I20260806 09:11:01.495388 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 9158
I20260806 09:11:01.496528 9242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:01.605976 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 9158
2026-08-06T09:11:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4442 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T09:11:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:01Z Disabled control of system clock
I20260806 09:11:01.687214 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42549
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:42549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 09:11:02.026619 9928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:02.026914 9928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:02.026999 9928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:02.034121 9928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:02.034318 9928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:02.034401 9928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:02.034526 9928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:02.034623 9928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:02.044486 9928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:42549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:02.045902 9928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:02.047924 9928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:02.060693 9936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:02.060742 9938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:02.060774 9935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:02.064831 9928 server_base.cc:1034] running on GCE node
I20260806 09:11:02.065676 9928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:02.067324 9928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:02.068622 9928 hybrid_clock.cc:648] HybridClock initialized: now 1786007462068568 us; error 65 us; skew 500 ppm
I20260806 09:11:02.069052 9928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:02.071563 9928 webserver.cc:460] Webserver started at http://127.3.69.62:43473/ using document root <none> and password file <none>
I20260806 09:11:02.072290 9928 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:02.072433 9928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:02.072794 9928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:02.075920 9928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "e7c399d0b4e940408ed6bd0df764ff08"
format_stamp: "Formatted at 2026-08-06 09:11:02 on dist-test-slave-w4v5"
server_key: "42063e101cf5dd4ce4b4c3da5c6df4e3"
server_key_iv: "f05523813bd9bbfb1512bfe6c190f129"
server_key_version: "encryptionkey@0"
I20260806 09:11:02.076768 9928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "e7c399d0b4e940408ed6bd0df764ff08"
format_stamp: "Formatted at 2026-08-06 09:11:02 on dist-test-slave-w4v5"
server_key: "42063e101cf5dd4ce4b4c3da5c6df4e3"
server_key_iv: "f05523813bd9bbfb1512bfe6c190f129"
server_key_version: "encryptionkey@0"
I20260806 09:11:02.082279 9928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:11:02.085929 9945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:02.087374 9928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:11:02.087607 9928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "e7c399d0b4e940408ed6bd0df764ff08"
format_stamp: "Formatted at 2026-08-06 09:11:02 on dist-test-slave-w4v5"
server_key: "42063e101cf5dd4ce4b4c3da5c6df4e3"
server_key_iv: "f05523813bd9bbfb1512bfe6c190f129"
server_key_version: "encryptionkey@0"
I20260806 09:11:02.087911 9928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:02.094061 9928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:02.094862 9928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:02.095257 9928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:02.114800 9928 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:42549
I20260806 09:11:02.114830 10014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:42549 every 8 connection(s)
I20260806 09:11:02.116297 9928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:02.118647 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 9928
I20260806 09:11:02.118858 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:02.119619 3348 external_mini_cluster.cc:1428] Setting key 682c143a36dff766ce9ee9f07647dec9
I20260806 09:11:02.120466 10015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:02.129367 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:02.121742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.3.69.62:42549)
0806 09:11:02.122882 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:02.122905 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:02.122959 (+ 54us) server_negotiation.cc:407] Connection header received
0806 09:11:02.125297 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:02.125326 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:02.125874 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:02.126326 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:02.127384 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:02.127393 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:02.127414 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:02.127433 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:02.127521 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:02.127542 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:02.127780 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:02.127908 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 09:11:02.128181 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":363,"threads_started":1}
I20260806 09:11:02.134748 10015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08: Bootstrap starting.
I20260806 09:11:02.137701 10015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:02.138680 10015 log.cc:824] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:02.141544 10015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08: No bootstrap required, opened a new log
I20260806 09:11:02.148681 10015 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c399d0b4e940408ed6bd0df764ff08" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42549 } }
I20260806 09:11:02.148986 10015 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:02.149066 10015 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c399d0b4e940408ed6bd0df764ff08, State: Initialized, Role: FOLLOWER
I20260806 09:11:02.149523 10015 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c399d0b4e940408ed6bd0df764ff08" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42549 } }
I20260806 09:11:02.149710 10015 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:02.149804 10015 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:02.149926 10015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:02.152122 10015 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c399d0b4e940408ed6bd0df764ff08" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42549 } }
I20260806 09:11:02.152719 10015 leader_election.cc:302] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c399d0b4e940408ed6bd0df764ff08; no voters:
I20260806 09:11:02.153318 10015 leader_election.cc:288] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:02.153429 10020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:02.154191 10020 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [term 1 LEADER]: Becoming Leader. State: Replica: e7c399d0b4e940408ed6bd0df764ff08, State: Running, Role: LEADER
I20260806 09:11:02.154778 10020 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c399d0b4e940408ed6bd0df764ff08" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42549 } }
I20260806 09:11:02.155495 10015 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:02.157377 10022 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c399d0b4e940408ed6bd0df764ff08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c399d0b4e940408ed6bd0df764ff08" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42549 } } }
I20260806 09:11:02.157709 10022 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:02.159432 10027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:02.159461 10023 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c399d0b4e940408ed6bd0df764ff08. Latest consensus state: current_term: 1 leader_uuid: "e7c399d0b4e940408ed6bd0df764ff08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c399d0b4e940408ed6bd0df764ff08" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42549 } } }
I20260806 09:11:02.159899 10023 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:02.164480 10027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:02.173621 10027 catalog_manager.cc:1347] Generated new cluster ID: c08ee16f95544579ac6addc77d52d530
I20260806 09:11:02.173784 10027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:02.192737 10027 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:02.193204 10027 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:02.200922 10027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08: Generated new TSK 0
I20260806 09:11:02.201767 10027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:11:02.212332 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42549
--builtin_ntp_servers=127.3.69.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 09:11:02.552497 10053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:02.552784 10053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:02.552953 10053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:02.560422 10053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:02.560523 10053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:02.560714 10053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:02.570135 10053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:02.571507 10053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:02.573351 10053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:02.586697 10059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:02.586725 10062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:02.587431 10060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:02.587633 10053 server_base.cc:1034] running on GCE node
I20260806 09:11:02.588326 10053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:02.589272 10053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:02.590487 10053 hybrid_clock.cc:648] HybridClock initialized: now 1786007462590436 us; error 36 us; skew 500 ppm
I20260806 09:11:02.591027 10053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:02.598022 10053 webserver.cc:460] Webserver started at http://127.3.69.1:44237/ using document root <none> and password file <none>
I20260806 09:11:02.598801 10053 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:02.598946 10053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:02.599363 10053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:02.602332 10053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "b256459045a741c3bbda8f7dd2e0f541"
format_stamp: "Formatted at 2026-08-06 09:11:02 on dist-test-slave-w4v5"
server_key: "180842dd5b065e696b02ebae9bfd57d2"
server_key_iv: "dcf1a8244292b032bef9f40cea43296d"
server_key_version: "encryptionkey@0"
I20260806 09:11:02.603204 10053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "b256459045a741c3bbda8f7dd2e0f541"
format_stamp: "Formatted at 2026-08-06 09:11:02 on dist-test-slave-w4v5"
server_key: "180842dd5b065e696b02ebae9bfd57d2"
server_key_iv: "dcf1a8244292b032bef9f40cea43296d"
server_key_version: "encryptionkey@0"
I20260806 09:11:02.608606 10053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:11:02.612139 10071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:02.613564 10053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:11:02.613751 10053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "b256459045a741c3bbda8f7dd2e0f541"
format_stamp: "Formatted at 2026-08-06 09:11:02 on dist-test-slave-w4v5"
server_key: "180842dd5b065e696b02ebae9bfd57d2"
server_key_iv: "dcf1a8244292b032bef9f40cea43296d"
server_key_version: "encryptionkey@0"
I20260806 09:11:02.614013 10053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:02.620002 10053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:02.620790 10053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:02.621080 10053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:02.622007 10053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:02.623483 10053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:02.623612 10053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:02.623749 10053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:02.623837 10053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:02.662911 10053 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:40879
I20260806 09:11:02.663010 10236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:40879 every 8 connection(s)
I20260806 09:11:02.664348 10053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:02.666184 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 10053
I20260806 09:11:02.666422 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:11:02.667368 3348 external_mini_cluster.cc:1428] Setting key 322268f7712c74434128c184b1d77df8
I20260806 09:11:02.670619 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42549
--builtin_ntp_servers=127.3.69.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 09:11:02.674185 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:02.667223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:40099 (local address 127.3.69.62:42549)
0806 09:11:02.668951 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:02.668957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:02.669700 (+ 743us) server_negotiation.cc:407] Connection header received
0806 09:11:02.671718 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:02.671723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:02.671793 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:02.671925 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:02.673346 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:02.673352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:02.673358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:02.673376 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:02.673431 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:02.673443 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:02.673575 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:02.673798 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 09:11:02.673872 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1574,"thread_start_us":1434,"threads_started":1}
I20260806 09:11:02.675159 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:02.668054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:42549 (local address 127.3.69.1:40099)
0806 09:11:02.669550 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:02.669586 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:02.671434 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:02.672197 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:02.672212 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:02.672319 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:02.673051 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:02.673054 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:02.673091 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:02.673176 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:02.673560 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:02.673571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:02.673580 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:11:02.673875 (+ 295us) client_negotiation.cc:240] Negotiation successful
0806 09:11:02.674131 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":700,"threads_started":1}
I20260806 09:11:02.676784 10237 heartbeater.cc:347] Connected to a master server at 127.3.69.62:42549
I20260806 09:11:02.677126 10237 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:02.677779 10237 heartbeater.cc:511] Master 127.3.69.62:42549 requested a full tablet report, sending...
I20260806 09:11:02.680039 9970 ts_manager.cc:194] Registered new tserver with Master: b256459045a741c3bbda8f7dd2e0f541 (127.3.69.1:40879)
W20260806 09:11:03.010406 10241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:03.010731 10241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:03.010878 10241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:03.018301 10241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:03.018394 10241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:03.018577 10241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:03.027779 10241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:03.029070 10241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:03.030838 10241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:03.043744 10248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:03.043644 10251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:03.043709 10249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:03.044806 10241 server_base.cc:1034] running on GCE node
I20260806 09:11:03.045388 10241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:03.046283 10241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:03.047509 10241 hybrid_clock.cc:648] HybridClock initialized: now 1786007463047463 us; error 59 us; skew 500 ppm
I20260806 09:11:03.047817 10241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:03.050062 10241 webserver.cc:460] Webserver started at http://127.3.69.2:36481/ using document root <none> and password file <none>
I20260806 09:11:03.050757 10241 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:03.050894 10241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:03.051288 10241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:03.054191 10241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "6beb507d8a394c68ab9747ffd75e697e"
format_stamp: "Formatted at 2026-08-06 09:11:03 on dist-test-slave-w4v5"
server_key: "6ea4109b724e316e96e2a7011bb9d2d1"
server_key_iv: "cb980d82fd70303b0407a0a9a8159645"
server_key_version: "encryptionkey@0"
I20260806 09:11:03.054965 10241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "6beb507d8a394c68ab9747ffd75e697e"
format_stamp: "Formatted at 2026-08-06 09:11:03 on dist-test-slave-w4v5"
server_key: "6ea4109b724e316e96e2a7011bb9d2d1"
server_key_iv: "cb980d82fd70303b0407a0a9a8159645"
server_key_version: "encryptionkey@0"
I20260806 09:11:03.060111 10241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:11:03.063581 10262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:03.064898 10241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:11:03.065104 10241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "6beb507d8a394c68ab9747ffd75e697e"
format_stamp: "Formatted at 2026-08-06 09:11:03 on dist-test-slave-w4v5"
server_key: "6ea4109b724e316e96e2a7011bb9d2d1"
server_key_iv: "cb980d82fd70303b0407a0a9a8159645"
server_key_version: "encryptionkey@0"
I20260806 09:11:03.065356 10241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:03.071025 10241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:03.071811 10241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:03.072093 10241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:03.073102 10241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:03.074537 10241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:03.074661 10241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:03.074815 10241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:03.074904 10241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:03.109961 10241 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:36295
I20260806 09:11:03.110013 10421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:36295 every 8 connection(s)
I20260806 09:11:03.111444 10241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:03.114382 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 10241
I20260806 09:11:03.114585 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:11:03.115137 3348 external_mini_cluster.cc:1428] Setting key 448e3ab158641b44bcc88d2b3193f8fb
I20260806 09:11:03.118517 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42549
--builtin_ntp_servers=127.3.69.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 09:11:03.120787 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.114048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:59179 (local address 127.3.69.62:42549)
0806 09:11:03.114279 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.114286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.116159 (+ 1873us) server_negotiation.cc:407] Connection header received
0806 09:11:03.118245 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.118251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.118312 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.118405 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.119985 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.119991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.119996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.120015 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.120058 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.120073 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.120189 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.120433 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.120501 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 09:11:03.121930 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.114685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:42549 (local address 127.3.69.2:59179)
0806 09:11:03.115962 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:03.116021 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:03.117939 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:03.118684 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:03.118698 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:03.118822 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:03.119614 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:03.119621 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:03.119673 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:03.119789 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:03.120195 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:03.120209 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:03.120221 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:11:03.120498 (+ 277us) client_negotiation.cc:240] Negotiation successful
0806 09:11:03.120744 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":458,"threads_started":1}
I20260806 09:11:03.123272 10422 heartbeater.cc:347] Connected to a master server at 127.3.69.62:42549
I20260806 09:11:03.123598 10422 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:03.124305 10422 heartbeater.cc:511] Master 127.3.69.62:42549 requested a full tablet report, sending...
I20260806 09:11:03.125893 9970 ts_manager.cc:194] Registered new tserver with Master: 6beb507d8a394c68ab9747ffd75e697e (127.3.69.2:36295)
W20260806 09:11:03.457765 10425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:03.458019 10425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:03.458110 10425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:03.465163 10425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:03.465257 10425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:03.465363 10425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:03.474460 10425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:03.475777 10425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:03.477425 10425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:03.490682 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:03.490710 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:03.490689 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:03.491647 10425 server_base.cc:1034] running on GCE node
I20260806 09:11:03.492306 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:03.493251 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:03.494460 10425 hybrid_clock.cc:648] HybridClock initialized: now 1786007463494414 us; error 53 us; skew 500 ppm
I20260806 09:11:03.494866 10425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:03.497756 10425 webserver.cc:460] Webserver started at http://127.3.69.3:36515/ using document root <none> and password file <none>
I20260806 09:11:03.498390 10425 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:03.498476 10425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:03.498741 10425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:03.501592 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "2e50e9a6aafd4042b62108de062cb07d"
format_stamp: "Formatted at 2026-08-06 09:11:03 on dist-test-slave-w4v5"
server_key: "26a495d46e88bf86234ff8d2bde9c8e5"
server_key_iv: "726ea5022edb2a1e72a8ff0f175ce9a4"
server_key_version: "encryptionkey@0"
I20260806 09:11:03.502499 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "2e50e9a6aafd4042b62108de062cb07d"
format_stamp: "Formatted at 2026-08-06 09:11:03 on dist-test-slave-w4v5"
server_key: "26a495d46e88bf86234ff8d2bde9c8e5"
server_key_iv: "726ea5022edb2a1e72a8ff0f175ce9a4"
server_key_version: "encryptionkey@0"
I20260806 09:11:03.507568 10425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 09:11:03.511148 10440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:03.512531 10425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:11:03.512739 10425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "2e50e9a6aafd4042b62108de062cb07d"
format_stamp: "Formatted at 2026-08-06 09:11:03 on dist-test-slave-w4v5"
server_key: "26a495d46e88bf86234ff8d2bde9c8e5"
server_key_iv: "726ea5022edb2a1e72a8ff0f175ce9a4"
server_key_version: "encryptionkey@0"
I20260806 09:11:03.513011 10425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:03.519482 10425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:03.520350 10425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:03.520658 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:03.521652 10425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:03.523222 10425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:03.523355 10425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:03.523504 10425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:03.523684 10425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:03.561131 10425 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:33165
I20260806 09:11:03.561192 10598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:33165 every 8 connection(s)
I20260806 09:11:03.562594 10425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:03.563300 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 10425
I20260806 09:11:03.563493 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:11:03.564121 3348 external_mini_cluster.cc:1428] Setting key 0c8ebffe44a295ac0965d2f897c3e2cf
I20260806 09:11:03.571962 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.565643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:40987 (local address 127.3.69.62:42549)
0806 09:11:03.565874 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.565881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.567844 (+ 1963us) server_negotiation.cc:407] Connection header received
0806 09:11:03.569686 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.569690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.569742 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.569829 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.571303 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.571307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.571310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.571325 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.571354 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.571365 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.571456 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.571691 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.571741 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:11:03.572934 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.566343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:42549 (local address 127.3.69.3:40987)
0806 09:11:03.567646 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:03.567696 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:03.569372 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:03.570134 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:03.570142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:03.570232 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:03.570965 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:03.570967 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:03.571005 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:03.571155 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:03.571486 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:03.571496 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:03.571502 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:11:03.571758 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 09:11:03.571951 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":369,"threads_started":1}
I20260806 09:11:03.574308 10599 heartbeater.cc:347] Connected to a master server at 127.3.69.62:42549
I20260806 09:11:03.574563 10599 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:03.575150 10599 heartbeater.cc:511] Master 127.3.69.62:42549 requested a full tablet report, sending...
I20260806 09:11:03.576452 9970 ts_manager.cc:194] Registered new tserver with Master: 2e50e9a6aafd4042b62108de062cb07d (127.3.69.3:33165)
I20260806 09:11:03.580518 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:11:03.587697 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.585146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.3.69.62:42549)
0806 09:11:03.585325 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.585329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.585848 (+ 519us) server_negotiation.cc:407] Connection header received
0806 09:11:03.586004 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.586008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.586075 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.586162 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.587177 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.587181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.587184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.587199 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.587229 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.587240 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.587340 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.587441 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.587488 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:11:03.591699 9970 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:11:03.598572 9970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:03.616117 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.610212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:33165 (local address 127.0.0.1:35122)
0806 09:11:03.611108 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:03.611156 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:03.611308 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:03.612621 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:03.612633 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:03.612743 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:03.613676 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:03.613679 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:03.613714 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:03.613792 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:03.615244 (+ 1452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:03.615260 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:03.615271 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:11:03.615460 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 09:11:03.615563 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":353,"threads_started":1}
I20260806 09:11:03.617028 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.610443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35122 (local address 127.3.69.3:33165)
0806 09:11:03.612008 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.612018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.612055 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:11:03.612138 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.612148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.612387 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.613273 (+ 886us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.614904 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.614913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.614929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.614953 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.615042 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.615065 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.616425 (+ 1360us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.616494 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.616603 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1405,"thread_start_us":1230,"threads_started":1}
I20260806 09:11:03.617636 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.611545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.3.69.2:36295)
0806 09:11:03.612548 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.612557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.613295 (+ 738us) server_negotiation.cc:407] Connection header received
0806 09:11:03.613533 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.613541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.613758 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.613884 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.616485 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.616494 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.616508 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.616539 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.616596 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.616619 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.616756 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.617110 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.617217 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":697,"threads_started":1}
I20260806 09:11:03.618386 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.610929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:36295 (local address 127.0.0.1:36692)
0806 09:11:03.613084 (+ 2155us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:03.613207 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:03.613381 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:03.615044 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:03.615048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:03.615075 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:03.616225 (+ 1150us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:03.616229 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:03.616245 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:03.616288 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:03.616858 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:03.616865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:03.616870 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 09:11:03.617906 (+ 1036us) client_negotiation.cc:240] Negotiation successful
0806 09:11:03.617958 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"thread_start_us":4807,"threads_started":1}
I20260806 09:11:03.620257 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.611928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.3.69.1:40879)
0806 09:11:03.617800 (+ 5872us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.617810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.617841 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:11:03.617913 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.617921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.618142 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.618356 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.619419 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.619427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.619438 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.619456 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.619513 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.619538 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.619686 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.619803 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.619910 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5706,"thread_start_us":5572,"threads_started":1}
I20260806 09:11:03.620359 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.611355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:40879 (local address 127.0.0.1:56178)
0806 09:11:03.615897 (+ 4542us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:03.615915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:03.616020 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:03.618367 (+ 2347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:03.618375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:03.618398 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:03.619199 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:03.619202 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:03.619218 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:03.619263 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:03.619670 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:03.619674 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:03.619679 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 09:11:03.619824 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 09:11:03.619874 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4404,"thread_start_us":1377,"threads_started":1}
I20260806 09:11:03.622498 10322 tablet_service.cc:1467] Processing CreateTablet for tablet 1c339475521a463c9c693deb4f0185b9 (DEFAULT_TABLE table=test-table [id=9b01332660f042dbb48b1cb66bb9bfd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:03.622491 10504 tablet_service.cc:1467] Processing CreateTablet for tablet 1c339475521a463c9c693deb4f0185b9 (DEFAULT_TABLE table=test-table [id=9b01332660f042dbb48b1cb66bb9bfd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:03.623915 10504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c339475521a463c9c693deb4f0185b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:03.624012 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c339475521a463c9c693deb4f0185b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:03.623927 10140 tablet_service.cc:1467] Processing CreateTablet for tablet 1c339475521a463c9c693deb4f0185b9 (DEFAULT_TABLE table=test-table [id=9b01332660f042dbb48b1cb66bb9bfd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:03.625231 10140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c339475521a463c9c693deb4f0185b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:03.635298 10622 tablet_bootstrap.cc:492] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Bootstrap starting.
I20260806 09:11:03.638881 10623 tablet_bootstrap.cc:492] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Bootstrap starting.
I20260806 09:11:03.639259 10622 tablet_bootstrap.cc:654] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:03.640260 10624 tablet_bootstrap.cc:492] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541: Bootstrap starting.
I20260806 09:11:03.640703 10622 log.cc:824] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:03.642108 10623 tablet_bootstrap.cc:654] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:03.643060 10623 log.cc:824] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:03.643826 10624 tablet_bootstrap.cc:654] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:03.644138 10622 tablet_bootstrap.cc:492] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: No bootstrap required, opened a new log
I20260806 09:11:03.644423 10622 ts_tablet_manager.cc:1397] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260806 09:11:03.645061 10624 log.cc:824] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:03.646193 10623 tablet_bootstrap.cc:492] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: No bootstrap required, opened a new log
I20260806 09:11:03.646468 10623 ts_tablet_manager.cc:1397] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260806 09:11:03.648059 10624 tablet_bootstrap.cc:492] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541: No bootstrap required, opened a new log
I20260806 09:11:03.648327 10624 ts_tablet_manager.cc:1397] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260806 09:11:03.653833 10623 raft_consensus.cc:348] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
I20260806 09:11:03.654203 10623 raft_consensus.cc:374] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:03.654307 10623 raft_consensus.cc:729] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6beb507d8a394c68ab9747ffd75e697e, State: Initialized, Role: FOLLOWER
I20260806 09:11:03.654946 10623 consensus_queue.cc:260] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
I20260806 09:11:03.654879 10622 raft_consensus.cc:348] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
I20260806 09:11:03.655331 10622 raft_consensus.cc:374] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:03.655429 10622 raft_consensus.cc:729] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e50e9a6aafd4042b62108de062cb07d, State: Initialized, Role: FOLLOWER
I20260806 09:11:03.656226 10422 heartbeater.cc:503] Master 127.3.69.62:42549 was elected leader, sending a full tablet report...
I20260806 09:11:03.656108 10622 consensus_queue.cc:260] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
I20260806 09:11:03.657267 10599 heartbeater.cc:503] Master 127.3.69.62:42549 was elected leader, sending a full tablet report...
I20260806 09:11:03.657904 10622 ts_tablet_manager.cc:1428] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260806 09:11:03.658841 10624 raft_consensus.cc:348] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
I20260806 09:11:03.659288 10624 raft_consensus.cc:374] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:03.659407 10624 raft_consensus.cc:729] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b256459045a741c3bbda8f7dd2e0f541, State: Initialized, Role: FOLLOWER
I20260806 09:11:03.659586 10623 ts_tablet_manager.cc:1428] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260806 09:11:03.660065 10624 consensus_queue.cc:260] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
I20260806 09:11:03.661269 10237 heartbeater.cc:503] Master 127.3.69.62:42549 was elected leader, sending a full tablet report...
I20260806 09:11:03.661726 10624 ts_tablet_manager.cc:1428] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260806 09:11:03.666731 10238 tablet.cc:2369] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:11:03.814702 10600 tablet.cc:2369] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:03.844061 10633 raft_consensus.cc:482] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:03.844444 10633 raft_consensus.cc:504] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
I20260806 09:11:03.846381 10633 leader_election.cc:288] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e50e9a6aafd4042b62108de062cb07d (127.3.69.3:33165), 6beb507d8a394c68ab9747ffd75e697e (127.3.69.2:36295)
I20260806 09:11:03.850090 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.846923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:49009 (local address 127.3.69.3:33165)
0806 09:11:03.847171 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.847180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.847963 (+ 783us) server_negotiation.cc:407] Connection header received
0806 09:11:03.848125 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.848131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.848198 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.848304 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.849398 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.849405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.849409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.849429 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.849469 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.849486 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.849612 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.849705 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.849754 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:11:03.850277 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.846799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:60989 (local address 127.3.69.2:36295)
0806 09:11:03.847169 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.847179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.848140 (+ 961us) server_negotiation.cc:407] Connection header received
0806 09:11:03.848253 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.848278 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.848344 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.848446 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.849617 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.849622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.849629 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.849652 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.849694 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.849715 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.849835 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.849938 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.850001 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260806 09:11:03.850318 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.846585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:33165 (local address 127.3.69.1:49009)
0806 09:11:03.847822 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:03.847844 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:03.847983 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:03.848353 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:03.848359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:03.848386 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:03.849184 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:03.849187 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:03.849201 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:03.849240 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:03.849589 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:03.849594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:03.849599 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 09:11:03.849745 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 09:11:03.849803 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":1071,"threads_started":1}
I20260806 09:11:03.850556 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.846850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:36295 (local address 127.3.69.1:60989)
0806 09:11:03.847980 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:03.848000 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:03.848144 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:03.848485 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:03.848488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:03.848509 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:03.849384 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:03.849388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:03.849403 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:03.849444 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:03.849819 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:03.849824 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:03.849828 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 09:11:03.849980 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 09:11:03.850026 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":539,"threads_started":1}
I20260806 09:11:03.851840 10533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c339475521a463c9c693deb4f0185b9" candidate_uuid: "b256459045a741c3bbda8f7dd2e0f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e50e9a6aafd4042b62108de062cb07d" is_pre_election: true
I20260806 09:11:03.851824 10353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c339475521a463c9c693deb4f0185b9" candidate_uuid: "b256459045a741c3bbda8f7dd2e0f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6beb507d8a394c68ab9747ffd75e697e" is_pre_election: true
I20260806 09:11:03.852391 10533 raft_consensus.cc:2454] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b256459045a741c3bbda8f7dd2e0f541 in term 0.
I20260806 09:11:03.852382 10353 raft_consensus.cc:2454] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b256459045a741c3bbda8f7dd2e0f541 in term 0.
I20260806 09:11:03.853012 10081 leader_election.cc:302] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6beb507d8a394c68ab9747ffd75e697e, b256459045a741c3bbda8f7dd2e0f541; no voters:
I20260806 09:11:03.853412 10633 raft_consensus.cc:2781] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:03.853569 10633 raft_consensus.cc:482] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:03.853678 10633 raft_consensus.cc:3037] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:03.855902 10633 raft_consensus.cc:504] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
I20260806 09:11:03.856513 10633 leader_election.cc:288] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [CANDIDATE]: Term 1 election: Requested vote from peers 2e50e9a6aafd4042b62108de062cb07d (127.3.69.3:33165), 6beb507d8a394c68ab9747ffd75e697e (127.3.69.2:36295)
I20260806 09:11:03.857002 10353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c339475521a463c9c693deb4f0185b9" candidate_uuid: "b256459045a741c3bbda8f7dd2e0f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6beb507d8a394c68ab9747ffd75e697e"
I20260806 09:11:03.856994 10533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c339475521a463c9c693deb4f0185b9" candidate_uuid: "b256459045a741c3bbda8f7dd2e0f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e50e9a6aafd4042b62108de062cb07d"
I20260806 09:11:03.857214 10353 raft_consensus.cc:3037] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:03.857214 10533 raft_consensus.cc:3037] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:03.860566 10533 raft_consensus.cc:2454] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b256459045a741c3bbda8f7dd2e0f541 in term 1.
I20260806 09:11:03.860556 10353 raft_consensus.cc:2454] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b256459045a741c3bbda8f7dd2e0f541 in term 1.
I20260806 09:11:03.860930 10081 leader_election.cc:302] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6beb507d8a394c68ab9747ffd75e697e, b256459045a741c3bbda8f7dd2e0f541; no voters:
I20260806 09:11:03.861231 10633 raft_consensus.cc:2781] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:03.861655 10633 raft_consensus.cc:686] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [term 1 LEADER]: Becoming Leader. State: Replica: b256459045a741c3bbda8f7dd2e0f541, State: Running, Role: LEADER
I20260806 09:11:03.862195 10633 consensus_queue.cc:237] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } }
W20260806 09:11:03.863516 10423 tablet.cc:2369] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:03.866963 9969 catalog_manager.cc:5495] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 reported cstate change: term changed from 0 to 1, leader changed from <none> to b256459045a741c3bbda8f7dd2e0f541 (127.3.69.1). New cstate: current_term: 1 leader_uuid: "b256459045a741c3bbda8f7dd2e0f541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b256459045a741c3bbda8f7dd2e0f541" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 40879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 } health_report { overall_health: UNKNOWN } } }
I20260806 09:11:03.919279 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.916702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.3.69.1:40879)
0806 09:11:03.916873 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.916877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.916900 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:11:03.917196 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.917201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.917317 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.917505 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.918501 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.918508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.918513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.918535 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.918579 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.918598 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.918722 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.918896 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.918947 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:11:03.927382 10533 raft_consensus.cc:1261] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 1 FOLLOWER]: Refusing update from remote peer b256459045a741c3bbda8f7dd2e0f541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:03.927382 10353 raft_consensus.cc:1261] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 1 FOLLOWER]: Refusing update from remote peer b256459045a741c3bbda8f7dd2e0f541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:03.928323 10640 consensus_queue.cc:1035] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e50e9a6aafd4042b62108de062cb07d" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:03.928648 10633 consensus_queue.cc:1035] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6beb507d8a394c68ab9747ffd75e697e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 36295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:03.946390 10645 mvcc.cc:205] Tried to move back new op lower bound from 7315486572235612160 to 7315486571982790656. Current Snapshot: MvccSnapshot[applied={T|T < 7315486572235612160}]
I20260806 09:11:03.954681 9969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:03.954989 9969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:03.958683 9969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7c399d0b4e940408ed6bd0df764ff08: Sending DeleteTablet for 3 replicas of tablet 1c339475521a463c9c693deb4f0185b9
I20260806 09:11:03.960023 10140 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c339475521a463c9c693deb4f0185b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:03 UTC) from {username='slave'} at 127.0.0.1:56178
I20260806 09:11:03.960089 10504 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c339475521a463c9c693deb4f0185b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:03 UTC) from {username='slave'} at 127.0.0.1:35122
I20260806 09:11:03.960851 10322 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c339475521a463c9c693deb4f0185b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:03 UTC) from {username='slave'} at 127.0.0.1:36692
I20260806 09:11:03.961864 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 10053
I20260806 09:11:03.979393 10661 tablet_replica.cc:330] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: stopping tablet replica
I20260806 09:11:03.980432 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:03.963618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.3.69.1:40879)
0806 09:11:03.963822 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:03.963831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:03.976055 (+ 12224us) server_negotiation.cc:407] Connection header received
0806 09:11:03.976662 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:03.976679 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:03.976763 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:03.976853 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:03.978619 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:03.978634 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:03.978676 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:03.978702 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:03.978769 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:03.978791 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:03.978953 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:03.979894 (+ 941us) server_negotiation.cc:299] Negotiation successful
0806 09:11:03.979975 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:11:03.981072 10660 tablet_replica.cc:330] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: stopping tablet replica
I20260806 09:11:03.981596 10660 raft_consensus.cc:2229] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:03.981828 10228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:03.982210 10660 raft_consensus.cc:2258] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:03.984339 10661 raft_consensus.cc:2229] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:03.984966 10661 raft_consensus.cc:2258] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:03.986176 10660 ts_tablet_manager.cc:1905] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:03.987378 10661 ts_tablet_manager.cc:1905] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:03.994081 10661 ts_tablet_manager.cc:1918] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:03.994257 10661 log.cc:1196] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/1c339475521a463c9c693deb4f0185b9
I20260806 09:11:03.994268 10660 ts_tablet_manager.cc:1918] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:03.994436 10660 log.cc:1196] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/1c339475521a463c9c693deb4f0185b9
I20260806 09:11:03.994822 10661 ts_tablet_manager.cc:1939] T 1c339475521a463c9c693deb4f0185b9 P 6beb507d8a394c68ab9747ffd75e697e: Deleting consensus metadata
I20260806 09:11:03.994987 10660 ts_tablet_manager.cc:1939] T 1c339475521a463c9c693deb4f0185b9 P 2e50e9a6aafd4042b62108de062cb07d: Deleting consensus metadata
I20260806 09:11:03.996466 9950 catalog_manager.cc:4841] TS 6beb507d8a394c68ab9747ffd75e697e (127.3.69.2:36295): tablet 1c339475521a463c9c693deb4f0185b9 (table test-table [id=9b01332660f042dbb48b1cb66bb9bfd5]) successfully deleted
I20260806 09:11:03.996824 9947 catalog_manager.cc:4841] TS 2e50e9a6aafd4042b62108de062cb07d (127.3.69.3:33165): tablet 1c339475521a463c9c693deb4f0185b9 (table test-table [id=9b01332660f042dbb48b1cb66bb9bfd5]) successfully deleted
I20260806 09:11:04.085301 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 10053
I20260806 09:11:04.085666 10659 tablet_replica.cc:330] T 1c339475521a463c9c693deb4f0185b9 P b256459045a741c3bbda8f7dd2e0f541: stopping tablet replica
W20260806 09:11:04.112483 9947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.69.1:40879 (error 108)
I20260806 09:11:04.113183 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 10241
I20260806 09:11:04.114639 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.113980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:40879 (local address 127.0.0.1:56194)
0806 09:11:04.114195 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:04.114413 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 09:11:04.115312 9947 catalog_manager.cc:4584] TS b256459045a741c3bbda8f7dd2e0f541 (127.3.69.1:40879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c339475521a463c9c693deb4f0185b9: Network error: Client connection negotiation failed: client connection to 127.3.69.1:40879: connect: Connection refused (error 111)
I20260806 09:11:04.125087 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.114298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.3.69.2:36295)
0806 09:11:04.114558 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:04.114564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:04.114587 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:11:04.114651 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:04.114657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:04.114817 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:04.114940 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:04.124340 (+ 9400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:04.124346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:04.124351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:04.124373 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:04.124416 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:04.124432 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:04.124596 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:04.124716 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 09:11:04.124777 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260806 09:11:04.125844 10412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:04.169185 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.168613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:40879 (local address 127.0.0.1:56206)
0806 09:11:04.168874 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:04.169005 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 09:11:04.170326 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.169888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:40879 (local address 127.0.0.1:56220)
0806 09:11:04.170070 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:04.170169 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 09:11:04.220674 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 10241
I20260806 09:11:04.231287 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.230615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:40879 (local address 127.0.0.1:56232)
0806 09:11:04.230894 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:04.231043 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 09:11:04.232532 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.232043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:40879 (local address 127.0.0.1:56234)
0806 09:11:04.232250 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:04.232365 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 09:11:04.244999 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 10425
I20260806 09:11:04.248148 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.245826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.3.69.3:33165)
0806 09:11:04.246042 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:04.246048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:04.246071 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:11:04.246227 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:04.246233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:04.246345 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:04.246459 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:04.247401 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:04.247407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:04.247412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:04.247431 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:04.247472 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:04.247484 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:04.247624 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:04.247778 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 09:11:04.247842 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:11:04.248842 10593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:04.313397 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.312827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:40879 (local address 127.0.0.1:56240)
0806 09:11:04.313092 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:04.313212 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 09:11:04.314599 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:04.314178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:40879 (local address 127.0.0.1:56244)
0806 09:11:04.314386 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:04.314461 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 09:11:04.346545 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 10425
I20260806 09:11:04.377132 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 9928
I20260806 09:11:04.378566 10008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:04.485144 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 9928
2026-08-06T09:11:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2877 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 09:11:04.535480 3348 mini_oidc.cc:152] Starting JWKS server
I20260806 09:11:04.539503 3348 webserver.cc:460] Webserver started at https://127.0.0.1:36247/ using document root <none> and password file <none>
I20260806 09:11:04.539783 3348 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:11:04.540365 3348 webserver.cc:460] Webserver started at http://127.0.0.1:36551/ using document root <none> and password file <none>
2026-08-06T09:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:04Z Disabled control of system clock
I20260806 09:11:04.572554 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37113
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:37113
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36247/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/testchainca.cert with env {}
W20260806 09:11:04.903679 10675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:04.904028 10675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:04.904204 10675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:04.911453 10675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:04.911594 10675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:04.911682 10675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:04.911795 10675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:04.911882 10675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:11:04.911974 10675 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:11:04.912037 10675 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36247/jwks.json
I20260806 09:11:04.921983 10675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:37113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36247/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:04.923346 10675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:04.924655 10675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:04.936760 10686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:04.936751 10683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:04.936807 10682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:04.937896 10675 server_base.cc:1034] running on GCE node
I20260806 09:11:04.938715 10675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:04.940258 10675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:04.941534 10675 hybrid_clock.cc:648] HybridClock initialized: now 1786007464941455 us; error 69 us; skew 500 ppm
I20260806 09:11:04.941929 10675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:04.944299 10675 webserver.cc:460] Webserver started at http://127.3.69.62:44789/ using document root <none> and password file <none>
I20260806 09:11:04.944977 10675 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:04.945117 10675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:04.945442 10675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:04.948412 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "46aa102dd0114905bdb1cd6555d8f20d"
format_stamp: "Formatted at 2026-08-06 09:11:04 on dist-test-slave-w4v5"
server_key: "1a8da3adec1ceaf32642283af8830798"
server_key_iv: "d315c81cf578c0e2c47ffd0cd41e3498"
server_key_version: "encryptionkey@0"
I20260806 09:11:04.949224 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "46aa102dd0114905bdb1cd6555d8f20d"
format_stamp: "Formatted at 2026-08-06 09:11:04 on dist-test-slave-w4v5"
server_key: "1a8da3adec1ceaf32642283af8830798"
server_key_iv: "d315c81cf578c0e2c47ffd0cd41e3498"
server_key_version: "encryptionkey@0"
I20260806 09:11:04.954368 10675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 09:11:04.957917 10693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:04.959411 10675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:04.959610 10675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "46aa102dd0114905bdb1cd6555d8f20d"
format_stamp: "Formatted at 2026-08-06 09:11:04 on dist-test-slave-w4v5"
server_key: "1a8da3adec1ceaf32642283af8830798"
server_key_iv: "d315c81cf578c0e2c47ffd0cd41e3498"
server_key_version: "encryptionkey@0"
I20260806 09:11:04.959839 10675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:04.983954 10675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:04.984808 10675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:04.985108 10675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:05.004561 10675 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:37113
I20260806 09:11:05.004581 10767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:37113 every 8 connection(s)
I20260806 09:11:05.005911 10675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:05.009640 10768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:05.012358 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 10675
I20260806 09:11:05.012629 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:05.013327 3348 external_mini_cluster.cc:1428] Setting key 30a78987c636c0d90c680210d2a92db2
I20260806 09:11:05.023345 10768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d: Bootstrap starting.
I20260806 09:11:05.027284 10768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:05.028573 10768 log.cc:824] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:05.029057 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:05.015518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.3.69.62:37113)
0806 09:11:05.016541 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:05.016555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:05.016606 (+ 51us) server_negotiation.cc:407] Connection header received
0806 09:11:05.018813 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:05.018846 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:05.019478 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:05.019915 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:05.020488 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.022444 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.023901 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.024882 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.025106 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:05.026222 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:05.026234 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:05.026253 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:05.026950 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:05.027070 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:05.027121 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:05.027253 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:05.027451 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 09:11:05.027808 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":312,"threads_started":1}
I20260806 09:11:05.031610 10768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d: No bootstrap required, opened a new log
I20260806 09:11:05.032780 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:11:05.039043 10768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46aa102dd0114905bdb1cd6555d8f20d" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37113 } }
I20260806 09:11:05.039489 10768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:05.039594 10768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46aa102dd0114905bdb1cd6555d8f20d, State: Initialized, Role: FOLLOWER
I20260806 09:11:05.040207 10768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46aa102dd0114905bdb1cd6555d8f20d" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37113 } }
I20260806 09:11:05.040447 10768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:05.040589 10768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:05.040793 10768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:05.043249 10768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46aa102dd0114905bdb1cd6555d8f20d" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37113 } }
I20260806 09:11:05.043742 10768 leader_election.cc:302] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46aa102dd0114905bdb1cd6555d8f20d; no voters:
I20260806 09:11:05.044306 10768 leader_election.cc:288] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:05.044459 10773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:05.045257 10773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [term 1 LEADER]: Becoming Leader. State: Replica: 46aa102dd0114905bdb1cd6555d8f20d, State: Running, Role: LEADER
I20260806 09:11:05.045840 10773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46aa102dd0114905bdb1cd6555d8f20d" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37113 } }
I20260806 09:11:05.046432 10768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:05.049543 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46aa102dd0114905bdb1cd6555d8f20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46aa102dd0114905bdb1cd6555d8f20d" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37113 } } }
I20260806 09:11:05.050040 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:05.050882 10777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46aa102dd0114905bdb1cd6555d8f20d. Latest consensus state: current_term: 1 leader_uuid: "46aa102dd0114905bdb1cd6555d8f20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46aa102dd0114905bdb1cd6555d8f20d" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37113 } } }
I20260806 09:11:05.051303 10777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:05.051780 10782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:05.057699 10782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:05.066447 10782 catalog_manager.cc:1347] Generated new cluster ID: ae70e287b3514be58059a423d40f4283
I20260806 09:11:05.066555 10782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:05.096398 10782 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:05.098237 10782 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:05.117401 10782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46aa102dd0114905bdb1cd6555d8f20d: Generated new TSK 0
I20260806 09:11:05.118512 10782 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 09:11:05.162006 10812 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 09:11:05.162348 10812 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:11:05.163476 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:05.141049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.3.69.62:37113)
0806 09:11:05.141228 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:05.141235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:05.141780 (+ 545us) server_negotiation.cc:407] Connection header received
0806 09:11:05.142102 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:05.142108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:05.142204 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:05.142356 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:11:05.142872 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.144041 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.145453 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.146145 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.146287 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:05.146811 (+ 524us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:11:05.162758 (+ 15947us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 09:11:05.163149 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46706: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 09:11:05.163609 10771 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46706: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 09:11:05.165323 10810 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:05.140939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37113 (local address 127.0.0.1:46706)
0806 09:11:05.141681 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:05.141697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:05.141836 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:05.142385 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:05.142388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:05.142398 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:11:05.142714 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:05.142722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.144190 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.144194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:05.145314 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:05.145324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.146271 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.146373 (+ 102us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:05.146498 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:05.146552 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:11:05.163032 (+ 16480us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 09:11:05.163437 (+ 405us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.69.62:37113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":398,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.69.62:37113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 09:11:05.167177 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 10675
I20260806 09:11:05.168340 10761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:05.262637 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 10675
2026-08-06T09:11:05Z chronyd exiting
I20260806 09:11:05.319392 3348 test_util.cc:175] -----------------------------------------------
I20260806 09:11:05.319586 3348 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786007418649204-3348-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (788 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 09:11:05.323514 3348 mini_oidc.cc:152] Starting JWKS server
I20260806 09:11:05.325397 3348 webserver.cc:460] Webserver started at https://127.0.0.1:34953/ using document root <none> and password file <none>
I20260806 09:11:05.325664 3348 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:11:05.326341 3348 webserver.cc:460] Webserver started at http://127.0.0.1:35097/ using document root <none> and password file <none>
2026-08-06T09:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:05Z Disabled control of system clock
I20260806 09:11:05.359625 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42845
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:42845
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34953/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/testchainca.cert with env {}
W20260806 09:11:05.689716 10821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:05.690052 10821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:05.690224 10821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:05.697348 10821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:05.697481 10821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:05.697578 10821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:05.697675 10821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:05.697739 10821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:11:05.697804 10821 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:11:05.697845 10821 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34953/jwks.json
I20260806 09:11:05.707329 10821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:42845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34953/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:05.708541 10821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:05.709761 10821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:05.721517 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:05.721552 10831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:05.721661 10834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:05.722553 10821 server_base.cc:1034] running on GCE node
I20260806 09:11:05.723412 10821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:05.724794 10821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:05.726058 10821 hybrid_clock.cc:648] HybridClock initialized: now 1786007465726014 us; error 53 us; skew 500 ppm
I20260806 09:11:05.726438 10821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:05.728854 10821 webserver.cc:460] Webserver started at http://127.3.69.62:37901/ using document root <none> and password file <none>
I20260806 09:11:05.729540 10821 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:05.729679 10821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:05.730008 10821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:05.732857 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "e54cad4a1f7b4e38884a4479e9bfe897"
format_stamp: "Formatted at 2026-08-06 09:11:05 on dist-test-slave-w4v5"
server_key: "9eb2552505eb7fd7e3d6b35325ddd85b"
server_key_iv: "b7a37f792635e00e281b155bfa47153f"
server_key_version: "encryptionkey@0"
I20260806 09:11:05.733644 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "e54cad4a1f7b4e38884a4479e9bfe897"
format_stamp: "Formatted at 2026-08-06 09:11:05 on dist-test-slave-w4v5"
server_key: "9eb2552505eb7fd7e3d6b35325ddd85b"
server_key_iv: "b7a37f792635e00e281b155bfa47153f"
server_key_version: "encryptionkey@0"
I20260806 09:11:05.738601 10821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 09:11:05.742029 10842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:05.743433 10821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:05.743626 10821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "e54cad4a1f7b4e38884a4479e9bfe897"
format_stamp: "Formatted at 2026-08-06 09:11:05 on dist-test-slave-w4v5"
server_key: "9eb2552505eb7fd7e3d6b35325ddd85b"
server_key_iv: "b7a37f792635e00e281b155bfa47153f"
server_key_version: "encryptionkey@0"
I20260806 09:11:05.743844 10821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:05.761389 10821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:05.762214 10821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:05.762511 10821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:05.781227 10821 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:42845
I20260806 09:11:05.781247 10922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:42845 every 8 connection(s)
I20260806 09:11:05.782624 10821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:05.786434 10923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:05.789637 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 10821
I20260806 09:11:05.789870 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:05.790740 3348 external_mini_cluster.cc:1428] Setting key b4987f0f2fc155fdc9fc99790ff7f271
I20260806 09:11:05.799208 10923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897: Bootstrap starting.
I20260806 09:11:05.802326 10923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:05.803504 10923 log.cc:824] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:05.805285 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:05.792823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.3.69.62:42845)
0806 09:11:05.793891 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:05.793903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:05.793947 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:11:05.795817 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:05.795844 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:05.796350 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:05.796813 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:05.797447 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.798838 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.800396 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.801164 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.801389 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:05.802724 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:05.802731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:05.802747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:05.803398 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:05.803506 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:05.803531 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:05.803685 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:05.803891 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 09:11:05.804278 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":359,"threads_started":1}
I20260806 09:11:05.806454 10923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897: No bootstrap required, opened a new log
I20260806 09:11:05.809445 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:11:05.816056 10923 raft_consensus.cc:348] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54cad4a1f7b4e38884a4479e9bfe897" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42845 } }
I20260806 09:11:05.816406 10923 raft_consensus.cc:374] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:05.816479 10923 raft_consensus.cc:729] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e54cad4a1f7b4e38884a4479e9bfe897, State: Initialized, Role: FOLLOWER
I20260806 09:11:05.817023 10923 consensus_queue.cc:260] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54cad4a1f7b4e38884a4479e9bfe897" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42845 } }
I20260806 09:11:05.817245 10923 raft_consensus.cc:388] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:05.817348 10923 raft_consensus.cc:482] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:05.817473 10923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:05.819742 10923 raft_consensus.cc:504] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54cad4a1f7b4e38884a4479e9bfe897" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42845 } }
I20260806 09:11:05.820345 10923 leader_election.cc:302] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e54cad4a1f7b4e38884a4479e9bfe897; no voters:
I20260806 09:11:05.820961 10923 leader_election.cc:288] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:05.821138 10930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:05.821875 10930 raft_consensus.cc:686] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [term 1 LEADER]: Becoming Leader. State: Replica: e54cad4a1f7b4e38884a4479e9bfe897, State: Running, Role: LEADER
I20260806 09:11:05.822407 10930 consensus_queue.cc:237] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54cad4a1f7b4e38884a4479e9bfe897" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42845 } }
I20260806 09:11:05.823220 10923 sys_catalog.cc:564] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:05.825258 10932 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e54cad4a1f7b4e38884a4479e9bfe897. Latest consensus state: current_term: 1 leader_uuid: "e54cad4a1f7b4e38884a4479e9bfe897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54cad4a1f7b4e38884a4479e9bfe897" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42845 } } }
I20260806 09:11:05.825635 10932 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:05.827298 10931 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e54cad4a1f7b4e38884a4479e9bfe897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54cad4a1f7b4e38884a4479e9bfe897" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42845 } } }
I20260806 09:11:05.827701 10931 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:05.828740 10937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:05.834280 10937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:05.843052 10937 catalog_manager.cc:1347] Generated new cluster ID: 76ce414ec6e547c98d81d500953998f8
I20260806 09:11:05.843199 10937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:05.868835 10937 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:05.870316 10937 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:05.889823 10937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e54cad4a1f7b4e38884a4479e9bfe897: Generated new TSK 0
I20260806 09:11:05.891024 10937 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 09:11:05.937525 10968 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 09:11:05.937677 10968 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:11:05.939067 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:05.916972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.3.69.62:42845)
0806 09:11:05.917182 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:05.917193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:05.917647 (+ 454us) server_negotiation.cc:407] Connection header received
0806 09:11:05.917850 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:05.917855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:05.917951 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:05.918113 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:11:05.918553 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.919747 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.921108 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.921857 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.921982 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:05.922475 (+ 493us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:11:05.938236 (+ 15761us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 09:11:05.938673 (+ 437us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 09:11:05.939254 10967 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:05.916879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:42845 (local address 127.0.0.1:37844)
0806 09:11:05.917546 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:05.917562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:05.917684 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:05.918144 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:05.918147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:05.918156 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:11:05.918407 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:05.918414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.919892 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.919896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:05.920960 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:05.920971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:05.921985 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:05.922025 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:05.922177 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:05.922210 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:11:05.938479 (+ 16269us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 09:11:05.938776 (+ 297us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.69.62:42845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":397,"threads_started":1}
W20260806 09:11:05.939270 10927 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260806 09:11:05.941262 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 10821
I20260806 09:11:05.942538 10915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:06.044909 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 10821
2026-08-06T09:11:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (784 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 09:11:06.108309 3348 mini_oidc.cc:152] Starting JWKS server
I20260806 09:11:06.110273 3348 webserver.cc:460] Webserver started at https://127.0.0.1:44643/ using document root <none> and password file <none>
I20260806 09:11:06.110559 3348 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:11:06.111316 3348 webserver.cc:460] Webserver started at http://127.0.0.1:38437/ using document root <none> and password file <none>
2026-08-06T09:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:06Z Disabled control of system clock
I20260806 09:11:06.143039 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42605
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:44411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:42605
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44643/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 09:11:06.468895 10978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:06.469172 10978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:06.469357 10978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:06.476325 10978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:06.476423 10978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:06.476493 10978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:06.476588 10978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:06.476675 10978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:11:06.476764 10978 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:11:06.476845 10978 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44643/jwks.json
I20260806 09:11:06.485527 10978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:44411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:42605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44643/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:06.486771 10978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:06.488577 10978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:06.500599 10988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:06.500552 10993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:06.500624 10989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:06.501847 10978 server_base.cc:1034] running on GCE node
I20260806 09:11:06.502576 10978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:06.503939 10978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:06.505196 10978 hybrid_clock.cc:648] HybridClock initialized: now 1786007466505130 us; error 48 us; skew 500 ppm
I20260806 09:11:06.505666 10978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:06.508595 10978 webserver.cc:460] Webserver started at http://127.3.69.62:43631/ using document root <none> and password file <none>
I20260806 09:11:06.509274 10978 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:06.509366 10978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:06.509636 10978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:06.512647 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "b57fab0532734b48ae8e2034e91fd010"
format_stamp: "Formatted at 2026-08-06 09:11:06 on dist-test-slave-w4v5"
server_key: "d4ba721d0715fb69470174cf6be65be8"
server_key_iv: "49cbbc6b9881b6704ba396fb8bc4ea88"
server_key_version: "encryptionkey@0"
I20260806 09:11:06.513402 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "b57fab0532734b48ae8e2034e91fd010"
format_stamp: "Formatted at 2026-08-06 09:11:06 on dist-test-slave-w4v5"
server_key: "d4ba721d0715fb69470174cf6be65be8"
server_key_iv: "49cbbc6b9881b6704ba396fb8bc4ea88"
server_key_version: "encryptionkey@0"
I20260806 09:11:06.518464 10978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:11:06.521870 11001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:06.523267 10978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 09:11:06.523406 10978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "b57fab0532734b48ae8e2034e91fd010"
format_stamp: "Formatted at 2026-08-06 09:11:06 on dist-test-slave-w4v5"
server_key: "d4ba721d0715fb69470174cf6be65be8"
server_key_iv: "49cbbc6b9881b6704ba396fb8bc4ea88"
server_key_version: "encryptionkey@0"
I20260806 09:11:06.523582 10978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:06.564096 10978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:06.564949 10978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:06.565217 10978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:06.584630 10978 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:42605
I20260806 09:11:06.584682 11079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:42605 every 8 connection(s)
I20260806 09:11:06.586046 10978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:06.589877 11080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:06.592671 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 10978
I20260806 09:11:06.592895 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:06.593720 3348 external_mini_cluster.cc:1428] Setting key fe9058372d3fd1436d2b5ee541cc71c2
I20260806 09:11:06.601764 11080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010: Bootstrap starting.
I20260806 09:11:06.604800 11080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:06.605818 11080 log.cc:824] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:06.607448 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:06.595720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.3.69.62:42605)
0806 09:11:06.596703 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:06.596714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:06.596755 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:11:06.598467 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:06.598490 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:06.598951 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:06.599397 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:06.599999 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:06.601225 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:06.602591 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:06.603354 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:06.603581 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:06.604742 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:06.604750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:06.604763 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:06.605363 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:06.605436 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:06.605461 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:06.605592 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:06.605832 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 09:11:06.606160 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":362,"threads_started":1}
I20260806 09:11:06.609061 11080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010: No bootstrap required, opened a new log
I20260806 09:11:06.611336 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:11:06.616858 11080 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57fab0532734b48ae8e2034e91fd010" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42605 } }
I20260806 09:11:06.617213 11080 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:06.617319 11080 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57fab0532734b48ae8e2034e91fd010, State: Initialized, Role: FOLLOWER
I20260806 09:11:06.617909 11080 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57fab0532734b48ae8e2034e91fd010" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42605 } }
I20260806 09:11:06.618173 11080 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:06.618331 11080 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:06.618530 11080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:06.620702 11080 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57fab0532734b48ae8e2034e91fd010" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42605 } }
I20260806 09:11:06.621186 11080 leader_election.cc:302] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57fab0532734b48ae8e2034e91fd010; no voters:
I20260806 09:11:06.621778 11080 leader_election.cc:288] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:06.621959 11087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:06.622682 11087 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [term 1 LEADER]: Becoming Leader. State: Replica: b57fab0532734b48ae8e2034e91fd010, State: Running, Role: LEADER
I20260806 09:11:06.623317 11087 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57fab0532734b48ae8e2034e91fd010" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42605 } }
I20260806 09:11:06.623996 11080 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:06.626454 11089 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57fab0532734b48ae8e2034e91fd010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57fab0532734b48ae8e2034e91fd010" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42605 } } }
I20260806 09:11:06.627038 11089 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:06.626427 11090 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57fab0532734b48ae8e2034e91fd010. Latest consensus state: current_term: 1 leader_uuid: "b57fab0532734b48ae8e2034e91fd010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57fab0532734b48ae8e2034e91fd010" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42605 } } }
I20260806 09:11:06.627487 11090 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:06.628213 11098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:06.633131 11098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:06.641986 11098 catalog_manager.cc:1347] Generated new cluster ID: e860b80c363642fd8a2501a32ecad952
I20260806 09:11:06.642113 11098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:06.666998 11098 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:06.669016 11098 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:06.681761 11098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57fab0532734b48ae8e2034e91fd010: Generated new TSK 0
I20260806 09:11:06.682634 11098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:11:06.715396 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:06.706887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.3.69.62:42605)
0806 09:11:06.707112 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:06.707120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:06.707515 (+ 395us) server_negotiation.cc:407] Connection header received
0806 09:11:06.707726 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:06.707734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:06.707824 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:06.707988 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:11:06.708445 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:06.709569 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:06.710936 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:06.711690 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:06.711816 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:06.712293 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:11:06.714708 (+ 2415us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44643/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 09:11:06.715042 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44643/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 09:11:06.715529 11084 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53284: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44643/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 09:11:06.717166 11115 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:06.706738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:42605 (local address 127.0.0.1:53284)
0806 09:11:06.707405 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:06.707426 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:06.707552 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:06.708015 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:06.708019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:06.708029 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:11:06.708304 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:06.708311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:06.709699 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:06.709703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:06.710794 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:06.710804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:06.711810 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:06.711872 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:06.712003 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:06.712055 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:11:06.714958 (+ 2903us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44643/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 09:11:06.716762 (+ 1804us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.69.62:42605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44643/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":1475,"thread_start_us":369,"threads_started":1}
I20260806 09:11:06.718158 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 10978
I20260806 09:11:06.719372 11073 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 09:11:06.727537 11117 webserver.cc:533] Webserver: sslize failed:
E20260806 09:11:06.727665 11117 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:11:06.831578 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 10978
2026-08-06T09:11:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (786 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T09:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:06Z Disabled control of system clock
I20260806 09:11:06.932835 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:44211
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:33205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:44211
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/testchainca.cert with env {}
W20260806 09:11:07.260636 11125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:07.260999 11125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:07.261161 11125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:07.268359 11125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:07.268513 11125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:07.268615 11125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:07.268734 11125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:07.268812 11125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:11:07.268903 11125 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:11:07.268981 11125 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 09:11:07.278846 11125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:44211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:07.280202 11125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:07.281529 11125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:07.293067 11136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:07.293111 11134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:07.293339 11132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:07.293955 11125 server_base.cc:1034] running on GCE node
I20260806 09:11:07.294782 11125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:07.296263 11125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:07.297492 11125 hybrid_clock.cc:648] HybridClock initialized: now 1786007467297455 us; error 48 us; skew 500 ppm
I20260806 09:11:07.297848 11125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:07.300171 11125 webserver.cc:460] Webserver started at http://127.3.69.62:37011/ using document root <none> and password file <none>
I20260806 09:11:07.300871 11125 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:07.301008 11125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:07.301335 11125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:07.304284 11125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "1901fc969f6d46148394958c10130d1f"
format_stamp: "Formatted at 2026-08-06 09:11:07 on dist-test-slave-w4v5"
server_key: "945e4241390ca03cb7bc1840265a0b9b"
server_key_iv: "ad739dd0463a61087c4e8c7557828e39"
server_key_version: "encryptionkey@0"
I20260806 09:11:07.305095 11125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "1901fc969f6d46148394958c10130d1f"
format_stamp: "Formatted at 2026-08-06 09:11:07 on dist-test-slave-w4v5"
server_key: "945e4241390ca03cb7bc1840265a0b9b"
server_key_iv: "ad739dd0463a61087c4e8c7557828e39"
server_key_version: "encryptionkey@0"
I20260806 09:11:07.310283 11125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:11:07.313942 11150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:07.315430 11125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:11:07.315625 11125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "1901fc969f6d46148394958c10130d1f"
format_stamp: "Formatted at 2026-08-06 09:11:07 on dist-test-slave-w4v5"
server_key: "945e4241390ca03cb7bc1840265a0b9b"
server_key_iv: "ad739dd0463a61087c4e8c7557828e39"
server_key_version: "encryptionkey@0"
I20260806 09:11:07.315856 11125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:07.338894 11125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:07.339803 11125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:07.340114 11125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:07.358904 11125 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:44211
I20260806 09:11:07.358912 11224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:44211 every 8 connection(s)
I20260806 09:11:07.360316 11125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:07.364070 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:07.364645 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 11125
I20260806 09:11:07.364852 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:07.365726 3348 external_mini_cluster.cc:1428] Setting key be74686b13268a169d96326a0c7021b1
I20260806 09:11:07.378032 11226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f: Bootstrap starting.
I20260806 09:11:07.379583 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:07.367713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.3.69.62:44211)
0806 09:11:07.368719 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:07.368729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:07.368770 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:11:07.370394 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:07.370416 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:07.370876 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:07.371274 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:07.371909 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:07.373154 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:07.375152 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:07.375850 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:07.375977 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:07.377221 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:07.377232 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:07.377248 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:07.377776 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:07.377855 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:07.377876 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:07.377966 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:07.378358 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 09:11:07.378600 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":402,"threads_started":1}
I20260806 09:11:07.381337 11226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:07.382552 11226 log.cc:824] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:07.383265 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:11:07.385226 11226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f: No bootstrap required, opened a new log
I20260806 09:11:07.392239 11226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901fc969f6d46148394958c10130d1f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44211 } }
I20260806 09:11:07.392587 11226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:07.392704 11226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1901fc969f6d46148394958c10130d1f, State: Initialized, Role: FOLLOWER
I20260806 09:11:07.393311 11226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901fc969f6d46148394958c10130d1f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44211 } }
I20260806 09:11:07.393530 11226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:07.393679 11226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:07.393851 11226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:07.396029 11226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901fc969f6d46148394958c10130d1f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44211 } }
I20260806 09:11:07.396533 11226 leader_election.cc:302] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1901fc969f6d46148394958c10130d1f; no voters:
I20260806 09:11:07.397089 11226 leader_election.cc:288] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:07.397246 11231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:07.398025 11231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [term 1 LEADER]: Becoming Leader. State: Replica: 1901fc969f6d46148394958c10130d1f, State: Running, Role: LEADER
I20260806 09:11:07.398568 11231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901fc969f6d46148394958c10130d1f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44211 } }
I20260806 09:11:07.399209 11226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:07.401605 11234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1901fc969f6d46148394958c10130d1f. Latest consensus state: current_term: 1 leader_uuid: "1901fc969f6d46148394958c10130d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901fc969f6d46148394958c10130d1f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44211 } } }
I20260806 09:11:07.401979 11234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:07.401651 11233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1901fc969f6d46148394958c10130d1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901fc969f6d46148394958c10130d1f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 44211 } } }
I20260806 09:11:07.402132 11233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:07.410792 11242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:07.416844 11242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:07.425534 11242 catalog_manager.cc:1347] Generated new cluster ID: 8149a40b451f44c48ce80b28d402ac0d
I20260806 09:11:07.425652 11242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:07.453306 11242 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:07.454600 11242 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:07.463138 11242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1901fc969f6d46148394958c10130d1f: Generated new TSK 0
I20260806 09:11:07.464136 11242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:11:07.940897 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:07.485315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.3.69.62:44211)
0806 09:11:07.485556 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:07.485562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:07.485914 (+ 352us) server_negotiation.cc:407] Connection header received
0806 09:11:07.486131 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:07.486136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:07.486215 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:07.486342 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:11:07.486930 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:07.488069 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:07.490075 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:07.490550 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:07.490651 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:07.491288 (+ 637us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:11:07.940076 (+448788us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 09:11:07.940585 (+ 509us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36456: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 09:11:07.941051 11229 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36456: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 09:11:07.941170 11267 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:07.485188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:44211 (local address 127.0.0.1:36456)
0806 09:11:07.485799 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:07.485825 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:07.485967 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:07.486390 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:07.486396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:07.486410 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:11:07.486729 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:07.486742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:07.488229 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:07.488235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:07.489923 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:07.489937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:07.490680 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:07.490746 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:07.490928 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:07.490974 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:11:07.940432 (+449458us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 09:11:07.940747 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.69.62:44211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":320,"threads_started":1}
I20260806 09:11:07.942641 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 11125
I20260806 09:11:07.943877 11220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:08.046665 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 11125
2026-08-06T09:11:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1196 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (420 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1528 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:11:10 dist-test-slave-w4v5 krb5kdc[11286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:11:10 dist-test-slave-w4v5 krb5kdc[11286](info): set up 2 sockets
Aug 06 09:11:10 dist-test-slave-w4v5 krb5kdc[11286](info): commencing operation
krb5kdc: starting...
W20260806 09:11:11.316057 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:11:11 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:11:11.557191 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:38983
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:35337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:38983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:11.891350 11307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:11.891709 11307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:11.891875 11307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:11.899010 11307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:11.899186 11307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:11.899294 11307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:11.899397 11307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:11.899473 11307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:11.909235 11307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:35337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:38983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:11.910578 11307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:11.911906 11307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:11.926204 11314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:11.926425 11318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:11.926776 11316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:11.927886 11307 server_base.cc:1034] running on GCE node
I20260806 09:11:11.928963 11307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:11.930819 11307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:11.932091 11307 hybrid_clock.cc:648] HybridClock initialized: now 1786007471932008 us; error 90 us; skew 500 ppm
I20260806 09:11:11.932477 11307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:11 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:11.935770 11307 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:11:11.937328 11307 webserver.cc:460] Webserver started at http://127.3.69.62:41087/ using document root <none> and password file <none>
I20260806 09:11:11.938051 11307 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:11.938202 11307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:11.938519 11307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:11.941738 11307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "b4a89e54c6e142b28a7f6c068e9529cb"
format_stamp: "Formatted at 2026-08-06 09:11:11 on dist-test-slave-w4v5"
server_key: "a4f3fb2e085f573483540302c2605725"
server_key_iv: "b6132532f8585317bf1a8940bfa8b102"
server_key_version: "encryptionkey@0"
I20260806 09:11:11.942494 11307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "b4a89e54c6e142b28a7f6c068e9529cb"
format_stamp: "Formatted at 2026-08-06 09:11:11 on dist-test-slave-w4v5"
server_key: "a4f3fb2e085f573483540302c2605725"
server_key_iv: "b6132532f8585317bf1a8940bfa8b102"
server_key_version: "encryptionkey@0"
I20260806 09:11:11.947588 11307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 09:11:11.950966 11331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:11.952298 11307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:11:11.952486 11307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "b4a89e54c6e142b28a7f6c068e9529cb"
format_stamp: "Formatted at 2026-08-06 09:11:11 on dist-test-slave-w4v5"
server_key: "a4f3fb2e085f573483540302c2605725"
server_key_iv: "b6132532f8585317bf1a8940bfa8b102"
server_key_version: "encryptionkey@0"
I20260806 09:11:11.952684 11307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:11.968715 11307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:11.972873 11307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:11.973166 11307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:11.994012 11307 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:38983
I20260806 09:11:11.994097 11396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:38983 every 8 connection(s)
I20260806 09:11:11.995494 11307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:11.999292 11401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:12.000404 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 11307
I20260806 09:11:12.000635 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:12.001320 3348 external_mini_cluster.cc:1428] Setting key 8ed9d10422757d1ea97e2928e84a7d0f
I20260806 09:11:12.013687 11401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb: Bootstrap starting.
Aug 06 09:11:12 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:12.017061 11401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:12.018493 11401 log.cc:824] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:12.022142 11401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb: No bootstrap required, opened a new log
I20260806 09:11:12.026458 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:12.003349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36190 (local address 127.3.69.62:38983)
0806 09:11:12.004224 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:12.004239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:12.004295 (+ 56us) server_negotiation.cc:407] Connection header received
0806 09:11:12.006041 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:12.006075 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:12.006581 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:12.007034 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:12.008405 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:12.010446 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:12.011880 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:12.012896 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:12.013094 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:12.017464 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:12.017476 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:12.017495 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:12.017557 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:12.021683 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:12.022415 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:12.022425 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:12.022433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:12.022679 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:12.023159 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:12.023165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:12.023168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:12.023883 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:12.024221 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:12.024588 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 09:11:12.025061 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":344,"threads_started":1}
I20260806 09:11:12.032006 11401 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a89e54c6e142b28a7f6c068e9529cb" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38983 } }
I20260806 09:11:12.032341 11401 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:12.032413 11401 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a89e54c6e142b28a7f6c068e9529cb, State: Initialized, Role: FOLLOWER
I20260806 09:11:12.032912 11401 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a89e54c6e142b28a7f6c068e9529cb" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38983 } }
I20260806 09:11:12.033120 11401 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:12.033221 11401 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:12.033349 11401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:12.035686 11401 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a89e54c6e142b28a7f6c068e9529cb" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38983 } }
I20260806 09:11:12.036237 11401 leader_election.cc:302] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4a89e54c6e142b28a7f6c068e9529cb; no voters:
I20260806 09:11:12.036839 11401 leader_election.cc:288] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:12.036976 11407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:12.037731 11407 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [term 1 LEADER]: Becoming Leader. State: Replica: b4a89e54c6e142b28a7f6c068e9529cb, State: Running, Role: LEADER
I20260806 09:11:12.038264 11407 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a89e54c6e142b28a7f6c068e9529cb" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38983 } }
I20260806 09:11:12.038950 11401 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:12.041007 11410 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4a89e54c6e142b28a7f6c068e9529cb. Latest consensus state: current_term: 1 leader_uuid: "b4a89e54c6e142b28a7f6c068e9529cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a89e54c6e142b28a7f6c068e9529cb" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38983 } } }
I20260806 09:11:12.041424 11410 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:12.041743 11409 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4a89e54c6e142b28a7f6c068e9529cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a89e54c6e142b28a7f6c068e9529cb" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38983 } } }
I20260806 09:11:12.042138 11409 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:12.044374 11414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:12.050117 11414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:12.059101 11414 catalog_manager.cc:1347] Generated new cluster ID: 86f9b55e2ab740d898eb9bfa9eeff3a6
I20260806 09:11:12.059231 11414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:12.097494 11414 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:12.098934 11414 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:12.118386 11414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4a89e54c6e142b28a7f6c068e9529cb: Generated new TSK 0
I20260806 09:11:12.119516 11414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:11:12.191216 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:38983
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:35337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:12.521878 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:12.522145 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:12.522238 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:12.530025 11440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:12.530248 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:12.530494 11440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:12.541637 11440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:35337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:12.542956 11440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:12.544480 11440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:12.557674 11446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:12.557731 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:12.557763 11448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:12.558663 11440 server_base.cc:1034] running on GCE node
I20260806 09:11:12.559351 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:12.560305 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:12.561517 11440 hybrid_clock.cc:648] HybridClock initialized: now 1786007472561474 us; error 31 us; skew 500 ppm
I20260806 09:11:12.561861 11440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:12 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:12.565119 11440 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:11:12.566571 11440 webserver.cc:460] Webserver started at http://127.3.69.1:43583/ using document root <none> and password file <none>
I20260806 09:11:12.567281 11440 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:12.567394 11440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:12.567664 11440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:12.570551 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "7c0c3f5ac155444aab8dcd0a7cb82a5e"
format_stamp: "Formatted at 2026-08-06 09:11:12 on dist-test-slave-w4v5"
server_key: "43dbabbaede34e9abbb927606be02a48"
server_key_iv: "0683f4935aa7227869a623ea757173e1"
server_key_version: "encryptionkey@0"
I20260806 09:11:12.571391 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "7c0c3f5ac155444aab8dcd0a7cb82a5e"
format_stamp: "Formatted at 2026-08-06 09:11:12 on dist-test-slave-w4v5"
server_key: "43dbabbaede34e9abbb927606be02a48"
server_key_iv: "0683f4935aa7227869a623ea757173e1"
server_key_version: "encryptionkey@0"
I20260806 09:11:12.576716 11440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 09:11:12.580065 11460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:12.581408 11440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:11:12.581633 11440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "7c0c3f5ac155444aab8dcd0a7cb82a5e"
format_stamp: "Formatted at 2026-08-06 09:11:12 on dist-test-slave-w4v5"
server_key: "43dbabbaede34e9abbb927606be02a48"
server_key_iv: "0683f4935aa7227869a623ea757173e1"
server_key_version: "encryptionkey@0"
I20260806 09:11:12.581853 11440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:12.610117 11440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:12.614049 11440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:12.614341 11440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:12.615320 11440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:12.616648 11440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:12.616755 11440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:12.616904 11440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:12.616977 11440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:12.651409 11440 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:45961
I20260806 09:11:12.651456 11625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:45961 every 8 connection(s)
I20260806 09:11:12.653093 11440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:12.654909 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 11440
I20260806 09:11:12.655154 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:11:12.655969 3348 external_mini_cluster.cc:1428] Setting key 69f18190c7c964b091930d4a41ca0062
Aug 06 09:11:12 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:12.673324 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:12.655725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:38171 (local address 127.3.69.62:38983)
0806 09:11:12.656255 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:12.656262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:12.659154 (+ 2892us) server_negotiation.cc:407] Connection header received
0806 09:11:12.659507 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:12.659512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:12.659590 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:12.659715 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:12.661179 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:12.661990 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:12.663599 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:12.664009 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:12.664156 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:12.667629 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:12.667634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:12.667641 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:12.667679 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:12.670242 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:12.670966 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:12.670972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:12.670975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:12.671161 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:12.671599 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:12.671605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:12.671608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:12.671882 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:12.672041 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:12.672725 (+ 684us) server_negotiation.cc:299] Negotiation successful
0806 09:11:12.672964 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260806 09:11:12.674252 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:12.656315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:38983 (local address 127.3.69.1:38171)
0806 09:11:12.657199 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:12.657259 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:12.659225 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:12.659946 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:12.659954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:12.660434 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:12.660986 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:12.661002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:12.662138 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:12.662144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:12.663452 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:12.663465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:12.664246 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:12.664255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:12.664485 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:12.665205 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:12.665227 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:12.667416 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:12.670384 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:12.670392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:12.670408 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:12.670835 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:12.671279 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:12.671284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:12.671288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:12.671476 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:12.672012 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:12.672019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:12.672332 (+ 313us) client_negotiation.cc:769] Sending connection context
0806 09:11:12.672590 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 09:11:12.673110 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":339,"threads_started":1}
I20260806 09:11:12.675911 11627 heartbeater.cc:347] Connected to a master server at 127.3.69.62:38983
I20260806 09:11:12.676292 11627 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:12.677245 11627 heartbeater.cc:511] Master 127.3.69.62:38983 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
I20260806 09:11:12.679656 11355 ts_manager.cc:194] Registered new tserver with Master: 7c0c3f5ac155444aab8dcd0a7cb82a5e (127.3.69.1:45961)
I20260806 09:11:12.681140 11355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:38171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:11:12.708405 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:38983
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:35337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:13.038389 11635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:13.038712 11635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:13.038874 11635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:13.046388 11635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:13.046542 11635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:13.046731 11635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:13.056718 11635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:35337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:13.058012 11635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:13.059455 11635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:13.072619 11641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:13.072690 11642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:13.072642 11646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:13.073807 11635 server_base.cc:1034] running on GCE node
I20260806 09:11:13.074447 11635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:13.075471 11635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:13.076718 11635 hybrid_clock.cc:648] HybridClock initialized: now 1786007473076679 us; error 55 us; skew 500 ppm
I20260806 09:11:13.077039 11635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.080291 11635 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:11:13.081621 11635 webserver.cc:460] Webserver started at http://127.3.69.2:39351/ using document root <none> and password file <none>
I20260806 09:11:13.082240 11635 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:13.082341 11635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:13.082590 11635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:13.085547 11635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "37fd4233117649fe9eee88b5787d7a14"
format_stamp: "Formatted at 2026-08-06 09:11:13 on dist-test-slave-w4v5"
server_key: "ce39fbb362479a4dc6e634e34db8f924"
server_key_iv: "a1378e33919df7ba75451f7114559b2f"
server_key_version: "encryptionkey@0"
I20260806 09:11:13.086485 11635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "37fd4233117649fe9eee88b5787d7a14"
format_stamp: "Formatted at 2026-08-06 09:11:13 on dist-test-slave-w4v5"
server_key: "ce39fbb362479a4dc6e634e34db8f924"
server_key_iv: "a1378e33919df7ba75451f7114559b2f"
server_key_version: "encryptionkey@0"
I20260806 09:11:13.091589 11635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:11:13.094818 11655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:13.096124 11635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:13.096295 11635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "37fd4233117649fe9eee88b5787d7a14"
format_stamp: "Formatted at 2026-08-06 09:11:13 on dist-test-slave-w4v5"
server_key: "ce39fbb362479a4dc6e634e34db8f924"
server_key_iv: "a1378e33919df7ba75451f7114559b2f"
server_key_version: "encryptionkey@0"
I20260806 09:11:13.096483 11635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:13.123139 11635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:13.126955 11635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:13.127264 11635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:13.128189 11635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:13.129477 11635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:13.129550 11635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:13.129724 11635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:13.129776 11635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:13.163604 11635 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:46669
I20260806 09:11:13.163661 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:46669 every 8 connection(s)
I20260806 09:11:13.164889 11635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:13.170141 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 11635
I20260806 09:11:13.170347 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:11:13.170933 3348 external_mini_cluster.cc:1428] Setting key e413d199486db067eccc1ec96792d30e
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:13.184862 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.167307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:45067 (local address 127.3.69.62:38983)
0806 09:11:13.167480 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.167485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.168625 (+ 1140us) server_negotiation.cc:407] Connection header received
0806 09:11:13.170433 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.170439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.170549 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.170654 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.172329 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.173058 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.174571 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.174994 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.175174 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.178704 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:13.178709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:13.178712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:13.178742 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:13.181623 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.182336 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.182343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.182349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.182522 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.182929 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.182937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.182943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.183344 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:13.183489 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:13.184145 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 09:11:13.184402 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 09:11:13.185478 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.167765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:38983 (local address 127.3.69.2:45067)
0806 09:11:13.168463 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:13.168505 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:13.170111 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:13.170862 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:13.170870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:13.171451 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:13.172153 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:13.172174 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.173220 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.173224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:13.174417 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:13.174429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.175244 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.175251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:13.175475 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.176144 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:13.176165 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:13.178484 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:13.181769 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:13.181776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:13.181786 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:13.182198 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:13.182625 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:13.182628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:13.182630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:13.182809 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:13.183475 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:13.183481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:13.183764 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 09:11:13.183981 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 09:11:13.184350 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":287,"threads_started":1}
I20260806 09:11:13.187187 11808 heartbeater.cc:347] Connected to a master server at 127.3.69.62:38983
I20260806 09:11:13.187548 11808 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:13.188301 11808 heartbeater.cc:511] Master 127.3.69.62:38983 requested a full tablet report, sending...
I20260806 09:11:13.189867 11355 ts_manager.cc:194] Registered new tserver with Master: 37fd4233117649fe9eee88b5787d7a14 (127.3.69.2:46669)
I20260806 09:11:13.190696 11355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:45067
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:11:13.226027 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:38983
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:35337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:13.555630 11813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:13.555999 11813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:13.556160 11813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:13.563442 11813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:13.563611 11813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:13.563802 11813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:13.573405 11813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:35337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:13.574673 11813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:13.576025 11813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:13.589342 11818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:13.589361 11821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:13.589390 11819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:13.590054 11813 server_base.cc:1034] running on GCE node
I20260806 09:11:13.590701 11813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:13.591733 11813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:13.592962 11813 hybrid_clock.cc:648] HybridClock initialized: now 1786007473592917 us; error 56 us; skew 500 ppm
I20260806 09:11:13.593281 11813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.596539 11813 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:11:13.597999 11813 webserver.cc:460] Webserver started at http://127.3.69.3:42247/ using document root <none> and password file <none>
I20260806 09:11:13.598681 11813 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:13.599025 11813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:13.599522 11813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:13.602532 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "c73cf5fe218e414799c25921e3c95420"
format_stamp: "Formatted at 2026-08-06 09:11:13 on dist-test-slave-w4v5"
server_key: "7e1586318ee49478faf77cfbe97b1b96"
server_key_iv: "755c850f7095519804c4eb13ca167f25"
server_key_version: "encryptionkey@0"
I20260806 09:11:13.603394 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "c73cf5fe218e414799c25921e3c95420"
format_stamp: "Formatted at 2026-08-06 09:11:13 on dist-test-slave-w4v5"
server_key: "7e1586318ee49478faf77cfbe97b1b96"
server_key_iv: "755c850f7095519804c4eb13ca167f25"
server_key_version: "encryptionkey@0"
I20260806 09:11:13.608621 11813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:11:13.611999 11833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:13.613287 11813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:11:13.613467 11813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "c73cf5fe218e414799c25921e3c95420"
format_stamp: "Formatted at 2026-08-06 09:11:13 on dist-test-slave-w4v5"
server_key: "7e1586318ee49478faf77cfbe97b1b96"
server_key_iv: "755c850f7095519804c4eb13ca167f25"
server_key_version: "encryptionkey@0"
I20260806 09:11:13.613670 11813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:13.628469 11813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:13.632944 11813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:13.633257 11813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:13.634197 11813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:13.635635 11813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:13.635743 11813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:13.635895 11813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:13.635968 11813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:13.671558 11813 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:43553
I20260806 09:11:13.671600 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:43553 every 8 connection(s)
I20260806 09:11:13.672806 11813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:13.679358 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 11813
I20260806 09:11:13.679543 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:11:13.679996 3348 external_mini_cluster.cc:1428] Setting key 543fac1ba4cebe52d0dd56d1c35131bc
I20260806 09:11:13.684224 11627 heartbeater.cc:503] Master 127.3.69.62:38983 was elected leader, sending a full tablet report...
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:13.694095 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.675377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:60761 (local address 127.3.69.62:38983)
0806 09:11:13.675599 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.675606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.676803 (+ 1197us) server_negotiation.cc:407] Connection header received
0806 09:11:13.678488 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.678494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.678567 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.678660 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.680356 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.681431 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.683882 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.684288 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.684472 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.688433 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:13.688438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:13.688441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:13.688475 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:13.691245 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.691916 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.691920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.691922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.692079 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.692502 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.692506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.692508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.692799 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:13.692931 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:13.693669 (+ 738us) server_negotiation.cc:299] Negotiation successful
0806 09:11:13.693843 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:11:13.695425 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.675890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:38983 (local address 127.3.69.3:60761)
0806 09:11:13.676650 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:13.676686 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:13.678232 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:13.678866 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:13.678874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:13.679455 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:13.680169 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:13.680187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.681638 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.681643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:13.683721 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:13.683735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.684600 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.684609 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:13.684842 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.685694 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:13.685734 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:13.688207 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:13.691369 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:13.691375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:13.691389 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:13.691807 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:13.692174 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:13.692177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:13.692180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:13.692353 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:13.692972 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:13.692978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:13.693300 (+ 322us) client_negotiation.cc:769] Sending connection context
0806 09:11:13.693550 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 09:11:13.694019 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":298,"threads_started":1}
I20260806 09:11:13.696863 11978 heartbeater.cc:347] Connected to a master server at 127.3.69.62:38983
I20260806 09:11:13.697134 11978 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:13.697803 11978 heartbeater.cc:511] Master 127.3.69.62:38983 requested a full tablet report, sending...
I20260806 09:11:13.699340 11355 ts_manager.cc:194] Registered new tserver with Master: c73cf5fe218e414799c25921e3c95420 (127.3.69.3:43553)
I20260806 09:11:13.700261 11355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:60761
I20260806 09:11:13.708477 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/test-admin.keytab.
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.740770 3348 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 09:11:13.746196 11990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:13.743444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.3.69.1:45961)
0806 09:11:13.744144 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.744152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.744183 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:11:13.744365 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.744372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.744582 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.744729 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.745765 (+ 1036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44950: BlockingRecv error: recv got EOF from 127.0.0.1:44950 (error 108)
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":336,"threads_started":1}
W20260806 09:11:13.746682 3348 init.cc:244] Error while opening credential cache '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5cc)
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.748224 3348 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.749538 3348 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:11:13.764950 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.751171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.3.69.1:45961)
0806 09:11:13.751365 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.751371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.751519 (+ 148us) server_negotiation.cc:407] Connection header received
0806 09:11:13.751658 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.751662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.751718 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.751797 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.752765 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.753753 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.754989 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.755605 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.755713 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.759840 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:13.759851 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:13.759865 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:13.759902 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:13.762502 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.763142 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.763149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.763151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.763273 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.763617 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.763621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.763623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.763972 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:13.764203 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:13.764480 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 09:11:13.764686 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 09:11:13.770632 11990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:13.768890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.3.69.1:45961)
0806 09:11:13.769112 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.769118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.769284 (+ 166us) server_negotiation.cc:407] Connection header received
0806 09:11:13.769459 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.769464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.769534 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.769636 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.770379 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44970: BlockingRecv error: recv got EOF from 127.0.0.1:44970 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 09:11:13.770902 3348 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5cc' of type FILE
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.772367 3348 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.773682 3348 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:11:13.787881 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.775049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.3.69.1:45961)
0806 09:11:13.775247 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.775252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.775464 (+ 212us) server_negotiation.cc:407] Connection header received
0806 09:11:13.775667 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.775672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.775738 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.775835 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.776757 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.777477 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.778671 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.779195 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.779325 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.782761 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:13.782767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:13.782771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:13.782824 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:13.785315 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.785956 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.785963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.785966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.786127 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.786533 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.786538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.786542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.786809 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:13.787082 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:13.787402 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 09:11:13.787603 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 09:11:13.792378 11990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:13.790568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.3.69.1:45961)
0806 09:11:13.790770 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.790776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.791045 (+ 269us) server_negotiation.cc:407] Connection header received
0806 09:11:13.791252 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.791257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.791336 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.791435 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.792141 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44998: BlockingRecv error: recv got EOF from 127.0.0.1:44998 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 09:11:13.792557 3348 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786007418649204-3348-0/krb5kdc/krb5cc' of type FILE
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.794054 3348 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:13.795463 3348 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:11:13.809147 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.796694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.3.69.1:45961)
0806 09:11:13.796894 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.796900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.797133 (+ 233us) server_negotiation.cc:407] Connection header received
0806 09:11:13.797321 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.797326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.797386 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.797483 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.798418 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.799212 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.800522 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.800934 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.801065 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.804414 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:13.804419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:13.804422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:13.804453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:13.806795 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.807489 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.807495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.807498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.807670 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.808006 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.808010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.808012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.808290 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:13.808574 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:13.808681 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 09:11:13.808850 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 09:11:13.810832 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 11440
I20260806 09:11:13.823434 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.811840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.3.69.1:45961)
0806 09:11:13.812056 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.812064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.812601 (+ 537us) server_negotiation.cc:407] Connection header received
0806 09:11:13.812716 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.812719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.812771 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.812844 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.813835 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.814475 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.815891 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.816298 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.816392 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.818502 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:13.818507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:13.818510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:13.818538 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:13.821017 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.821598 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.821602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.821604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.821729 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.822090 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.822093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.822095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.822336 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:13.822532 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:13.822891 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 09:11:13.823066 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:11:13.824118 11619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:13.908806 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 11440
I20260806 09:11:13.933892 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 11635
Aug 06 09:11:13 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:11:13.954412 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:13.934799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.3.69.2:46669)
0806 09:11:13.935429 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:13.935436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:13.935462 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:11:13.935525 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:13.935531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:13.935746 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:13.935878 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:13.937331 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.939233 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.941454 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:13.942371 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:13.942476 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:13.947866 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:13.947877 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:13.947891 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:13.947926 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:13.951332 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.952032 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.952042 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.952046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.952194 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:13.952679 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:13.952682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:13.952684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:13.953024 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:13.953293 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:13.953796 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 09:11:13.954090 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":361,"threads_started":1}
I20260806 09:11:13.955941 11796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:14.045727 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 11635
I20260806 09:11:14.067924 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 11813
Aug 06 09:11:14 dist-test-slave-w4v5 krb5kdc[11286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:11:14.083729 12010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:14.068949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.3.69.3:43553)
0806 09:11:14.069542 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:14.069549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:14.069578 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:11:14.069649 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:14.069655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:14.069870 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:14.069998 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:14.071024 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:14.072128 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:14.073385 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:14.073978 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:14.074099 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:14.078237 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:14.078249 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:14.078264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:14.078298 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:14.081090 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:14.081738 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:14.081745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:14.081747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:14.081866 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:14.082206 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:14.082210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:14.082212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:14.082560 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:14.082813 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:14.083044 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 09:11:14.083299 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":285,"threads_started":1}
I20260806 09:11:14.085001 11972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:14.168129 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 11813
I20260806 09:11:14.192795 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 11307
I20260806 09:11:14.194132 11392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:14.300729 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 11307
2026-08-06T09:11:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4309 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:11:14 dist-test-slave-w4v5 krb5kdc[12015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:11:14 dist-test-slave-w4v5 krb5kdc[12015](info): set up 2 sockets
Aug 06 09:11:14 dist-test-slave-w4v5 krb5kdc[12015](info): commencing operation
krb5kdc: starting...
W20260806 09:11:15.608295 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:11:15 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.62.keytab.
Entry for principal oryx/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.62.keytab.
I20260806 09:11:15.851478 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:46261
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.62.keytab
--principal=oryx/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:46261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:16.179144 12034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:16.179481 12034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:16.179654 12034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:16.186753 12034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:16.186901 12034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:16.187013 12034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:16.187142 12034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:16.187233 12034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:16.196992 12034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:46261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.62.keytab
--principal=oryx/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:46261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:16.198348 12034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:16.199802 12034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:16.211627 12039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:16.211544 12040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:16.211601 12043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:16.212918 12034 server_base.cc:1034] running on GCE node
I20260806 09:11:16.213778 12034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:16.215214 12034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:16.216442 12034 hybrid_clock.cc:648] HybridClock initialized: now 1786007476216404 us; error 45 us; skew 500 ppm
I20260806 09:11:16.216800 12034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:16 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007476, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:16.222879 12034 init.cc:377] Logged in from keytab as oryx/127.3.69.62@KRBTEST.COM (short username oryx)
I20260806 09:11:16.224404 12034 webserver.cc:460] Webserver started at http://127.3.69.62:40187/ using document root <none> and password file <none>
I20260806 09:11:16.225099 12034 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:16.225255 12034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:16.225564 12034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:16.228557 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "7f4dfd6248ab451497efc55c64e34050"
format_stamp: "Formatted at 2026-08-06 09:11:16 on dist-test-slave-w4v5"
server_key: "d3760c90f380db729492b1ab2c0d10b1"
server_key_iv: "dc402d62572649cc2506b4d45c533872"
server_key_version: "encryptionkey@0"
I20260806 09:11:16.229313 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "7f4dfd6248ab451497efc55c64e34050"
format_stamp: "Formatted at 2026-08-06 09:11:16 on dist-test-slave-w4v5"
server_key: "d3760c90f380db729492b1ab2c0d10b1"
server_key_iv: "dc402d62572649cc2506b4d45c533872"
server_key_version: "encryptionkey@0"
I20260806 09:11:16.234453 12034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 09:11:16.238181 12053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:16.239533 12034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:16.239753 12034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "7f4dfd6248ab451497efc55c64e34050"
format_stamp: "Formatted at 2026-08-06 09:11:16 on dist-test-slave-w4v5"
server_key: "d3760c90f380db729492b1ab2c0d10b1"
server_key_iv: "dc402d62572649cc2506b4d45c533872"
server_key_version: "encryptionkey@0"
I20260806 09:11:16.239946 12034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:16.260317 12034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:16.264415 12034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:16.264766 12034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:16.283775 12034 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:46261
I20260806 09:11:16.283826 12128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:46261 every 8 connection(s)
I20260806 09:11:16.285208 12034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:16.289088 12130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:16.294740 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 12034
I20260806 09:11:16.294948 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:16.295817 3348 external_mini_cluster.cc:1428] Setting key f95c26bad9aaf158beb89b8106273a9b
I20260806 09:11:16.301227 12130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: Bootstrap starting.
I20260806 09:11:16.305173 12130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:16.306457 12130 log.cc:824] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:16.310099 12130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: No bootstrap required, opened a new log
Aug 06 09:11:16 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:16.318089 12130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } }
I20260806 09:11:16.318499 12130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:16.318605 12130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4dfd6248ab451497efc55c64e34050, State: Initialized, Role: FOLLOWER
I20260806 09:11:16.319497 12130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } }
I20260806 09:11:16.319792 12130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:16.319952 12130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:16.320161 12130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:16.320861 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:16.298050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.3.69.62:46261)
0806 09:11:16.298932 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:16.298944 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:16.298990 (+ 46us) server_negotiation.cc:407] Connection header received
0806 09:11:16.300799 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:16.300821 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:16.301211 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:16.301583 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:16.303505 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:16.304809 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:16.306652 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:16.307754 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:16.307971 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:16.312841 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:16.312853 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:16.312871 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:16.312918 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:16.316249 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:16.316981 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:16.316987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:16.316999 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:16.317176 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:16.317574 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:16.317579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:16.317584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:16.318130 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:16.318444 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:16.318919 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 09:11:16.319402 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":409,"threads_started":1}
I20260806 09:11:16.322760 12130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } }
I20260806 09:11:16.323304 12130 leader_election.cc:302] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f4dfd6248ab451497efc55c64e34050; no voters:
I20260806 09:11:16.324095 12130 leader_election.cc:288] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:16.324206 12136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:16.324931 12136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 1 LEADER]: Becoming Leader. State: Replica: 7f4dfd6248ab451497efc55c64e34050, State: Running, Role: LEADER
I20260806 09:11:16.325495 12136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } }
I20260806 09:11:16.326229 12130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:16.328527 12137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } }
I20260806 09:11:16.328984 12137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:16.328953 12138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f4dfd6248ab451497efc55c64e34050. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } }
I20260806 09:11:16.329285 12138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:16.329973 12147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:16.338697 12147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:16.347580 12147 catalog_manager.cc:1347] Generated new cluster ID: e1c2b4515b97401c906286923abfccca
I20260806 09:11:16.347728 12147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:16.388736 12147 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:16.390054 12147 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:16.396685 12147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: Generated new TSK 0
I20260806 09:11:16.397535 12147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab.
Entry for principal oryx/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab.
I20260806 09:11:16.467535 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:46261
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab
--principal=oryx/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:16.889382 12166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:16.889731 12166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:16.889899 12166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:16.897815 12166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:16.897976 12166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:16.898151 12166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:16.908443 12166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab
--principal=oryx/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:16.909878 12166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:16.911372 12166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:16.928264 12176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:16.928563 12179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:16.929143 12177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:16.929642 12166 server_base.cc:1034] running on GCE node
I20260806 09:11:16.930307 12166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:16.931540 12166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:16.932770 12166 hybrid_clock.cc:648] HybridClock initialized: now 1786007476932702 us; error 57 us; skew 500 ppm
I20260806 09:11:16.933115 12166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:16 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007476, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:16.937028 12166 init.cc:377] Logged in from keytab as oryx/127.3.69.1@KRBTEST.COM (short username oryx)
I20260806 09:11:16.942108 12166 webserver.cc:460] Webserver started at http://127.3.69.1:35805/ using document root <none> and password file <none>
I20260806 09:11:16.943239 12166 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:16.943408 12166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:16.943849 12166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:16.948865 12166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "c59aeb9db55843f8af2e0a3197e5ea96"
format_stamp: "Formatted at 2026-08-06 09:11:16 on dist-test-slave-w4v5"
server_key: "ffe7ad70fe4f957d14d3d23819e272c0"
server_key_iv: "a1424b879b15e61ec13d78910e8cc4dd"
server_key_version: "encryptionkey@0"
I20260806 09:11:16.950110 12166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "c59aeb9db55843f8af2e0a3197e5ea96"
format_stamp: "Formatted at 2026-08-06 09:11:16 on dist-test-slave-w4v5"
server_key: "ffe7ad70fe4f957d14d3d23819e272c0"
server_key_iv: "a1424b879b15e61ec13d78910e8cc4dd"
server_key_version: "encryptionkey@0"
I20260806 09:11:16.958068 12166 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260806 09:11:16.962905 12187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:16.964885 12166 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:11:16.965090 12166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "c59aeb9db55843f8af2e0a3197e5ea96"
format_stamp: "Formatted at 2026-08-06 09:11:16 on dist-test-slave-w4v5"
server_key: "ffe7ad70fe4f957d14d3d23819e272c0"
server_key_iv: "a1424b879b15e61ec13d78910e8cc4dd"
server_key_version: "encryptionkey@0"
I20260806 09:11:16.965363 12166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:16.993568 12166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:17.000092 12166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:17.000486 12166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:17.001971 12166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:17.004108 12166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:17.004217 12166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:17.004359 12166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:17.004433 12166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:17.042089 12166 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:45171
I20260806 09:11:17.042137 12334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:45171 every 8 connection(s)
I20260806 09:11:17.043537 12166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:17.044972 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 12166
I20260806 09:11:17.045209 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:11:17.045871 3348 external_mini_cluster.cc:1428] Setting key d5cd875ad465bf573ef9f81233c858ea
Aug 06 09:11:17 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007476, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:17.067261 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:17.046346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:44235 (local address 127.3.69.62:46261)
0806 09:11:17.047178 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:17.047185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:17.048050 (+ 865us) server_negotiation.cc:407] Connection header received
0806 09:11:17.050979 (+ 2929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:17.050983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:17.051062 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:17.051239 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:17.053377 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:17.054114 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:17.056664 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:17.057061 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:17.057165 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:17.061391 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:17.061396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:17.061399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:17.061433 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:17.064096 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:17.064867 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:17.064879 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:17.064882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:17.065001 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:17.065473 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:17.065477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:17.065479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:17.065741 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:17.065899 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:17.066615 (+ 716us) server_negotiation.cc:299] Negotiation successful
0806 09:11:17.066849 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":529,"threads_started":1}
I20260806 09:11:17.068269 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:17.046951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.1:44235)
0806 09:11:17.047902 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:17.047958 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:17.050655 (+ 2697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:17.051512 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:17.051535 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:17.052333 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:17.053184 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:17.053212 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:17.054290 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:17.054299 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:17.056479 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:17.056497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:17.057297 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:17.057307 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:17.057657 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:17.058385 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:17.058413 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:17.061160 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:17.064263 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:17.064272 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:17.064289 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:17.064731 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:17.065137 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:17.065142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:17.065145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:17.065340 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:17.065874 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:17.065882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:17.066232 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 09:11:17.066503 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 09:11:17.067055 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":340,"threads_started":1}
I20260806 09:11:17.069947 12335 heartbeater.cc:347] Connected to a master server at 127.3.69.62:46261
I20260806 09:11:17.070264 12335 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:17.071237 12335 heartbeater.cc:511] Master 127.3.69.62:46261 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.69.2@KRBTEST.COM; defaulting to no policy
I20260806 09:11:17.073675 12076 ts_manager.cc:194] Registered new tserver with Master: c59aeb9db55843f8af2e0a3197e5ea96 (127.3.69.1:45171)
I20260806 09:11:17.075227 12076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:44235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab.
Entry for principal oryx/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab.
I20260806 09:11:17.106122 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:46261
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab
--principal=oryx/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:17.448535 12345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:17.448791 12345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:17.448881 12345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:17.455794 12345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:17.455890 12345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:17.455994 12345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:17.465266 12345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab
--principal=oryx/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:17.466459 12345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:17.467746 12345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:17.481244 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:17.481388 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:17.481345 12354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:17.482304 12345 server_base.cc:1034] running on GCE node
I20260806 09:11:17.482940 12345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:17.483922 12345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:17.485160 12345 hybrid_clock.cc:648] HybridClock initialized: now 1786007477485125 us; error 50 us; skew 500 ppm
I20260806 09:11:17.485478 12345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:17 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007477, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:17.488880 12345 init.cc:377] Logged in from keytab as oryx/127.3.69.2@KRBTEST.COM (short username oryx)
I20260806 09:11:17.490240 12345 webserver.cc:460] Webserver started at http://127.3.69.2:45553/ using document root <none> and password file <none>
I20260806 09:11:17.490885 12345 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:17.490983 12345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:17.491309 12345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:17.494110 12345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "e4d35582e97a4bfea3165b8f33a3606b"
format_stamp: "Formatted at 2026-08-06 09:11:17 on dist-test-slave-w4v5"
server_key: "f20f7a9705dc60f4b227f6e30c552931"
server_key_iv: "9775b96ffb739921b97397a5cc7855d3"
server_key_version: "encryptionkey@0"
I20260806 09:11:17.495009 12345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "e4d35582e97a4bfea3165b8f33a3606b"
format_stamp: "Formatted at 2026-08-06 09:11:17 on dist-test-slave-w4v5"
server_key: "f20f7a9705dc60f4b227f6e30c552931"
server_key_iv: "9775b96ffb739921b97397a5cc7855d3"
server_key_version: "encryptionkey@0"
I20260806 09:11:17.500197 12345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:11:17.503527 12363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:17.504933 12345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:11:17.505101 12345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "e4d35582e97a4bfea3165b8f33a3606b"
format_stamp: "Formatted at 2026-08-06 09:11:17 on dist-test-slave-w4v5"
server_key: "f20f7a9705dc60f4b227f6e30c552931"
server_key_iv: "9775b96ffb739921b97397a5cc7855d3"
server_key_version: "encryptionkey@0"
I20260806 09:11:17.505306 12345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:17.526126 12345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:17.530179 12345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:17.530486 12345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:17.531520 12345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:17.532923 12345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:17.532995 12345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:17.533166 12345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:17.533217 12345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:17.568717 12345 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:40105
I20260806 09:11:17.568810 12516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:40105 every 8 connection(s)
I20260806 09:11:17.570107 12345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:17.579974 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 12345
I20260806 09:11:17.580168 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:11:17.580806 3348 external_mini_cluster.cc:1428] Setting key d82550bd2ff64ade980ddcc9267f031b
Aug 06 09:11:17 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007477, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:17.592878 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:17.572877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:43361 (local address 127.3.69.62:46261)
0806 09:11:17.573371 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:17.573377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:17.574153 (+ 776us) server_negotiation.cc:407] Connection header received
0806 09:11:17.576084 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:17.576090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:17.576191 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:17.576316 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:17.578226 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:17.579031 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:17.581333 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:17.581984 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:17.582088 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:17.586027 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:17.586035 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:17.586043 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:17.586089 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:17.589768 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:17.590509 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:17.590518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:17.590523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:17.590686 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:17.591135 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:17.591143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:17.591148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:17.591521 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:17.591686 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:17.592329 (+ 643us) server_negotiation.cc:299] Negotiation successful
0806 09:11:17.592583 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260806 09:11:17.594161 12520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:17.573258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.2:43361)
0806 09:11:17.573982 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:17.574020 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:17.575813 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:17.576553 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:17.576562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:17.577059 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:17.577982 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:17.578005 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:17.579234 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:17.579240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:17.581180 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:17.581195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:17.582221 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:17.582229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:17.582518 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:17.583301 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:17.583328 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:17.585782 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:17.589929 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:17.589936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:17.589948 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:17.590375 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:17.590808 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:17.590812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:17.590814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:17.591006 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:17.591648 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:17.591654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:17.591955 (+ 301us) client_negotiation.cc:769] Sending connection context
0806 09:11:17.592199 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 09:11:17.592651 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":295,"threads_started":1}
I20260806 09:11:17.595625 12517 heartbeater.cc:347] Connected to a master server at 127.3.69.62:46261
I20260806 09:11:17.595933 12517 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:17.596632 12517 heartbeater.cc:511] Master 127.3.69.62:46261 requested a full tablet report, sending...
I20260806 09:11:17.598119 12076 ts_manager.cc:194] Registered new tserver with Master: e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2:40105)
I20260806 09:11:17.598919 12076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:43361
WARNING: no policy specified for oryx/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab.
Entry for principal oryx/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab.
I20260806 09:11:17.634305 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:46261
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab
--principal=oryx/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:17.961774 12527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:17.962049 12527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:17.962144 12527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:17.969238 12527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:17.969339 12527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:17.969449 12527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:17.978896 12527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab
--principal=oryx/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:46261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:17.980101 12527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:17.981314 12527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:17.998955 12535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:17.998893 12538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:17.998932 12536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:18.000924 12527 server_base.cc:1034] running on GCE node
I20260806 09:11:18.001534 12527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:18.002439 12527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:18.003636 12527 hybrid_clock.cc:648] HybridClock initialized: now 1786007478003588 us; error 45 us; skew 500 ppm
I20260806 09:11:18.003969 12527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:17 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007477, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:18.007299 12527 init.cc:377] Logged in from keytab as oryx/127.3.69.3@KRBTEST.COM (short username oryx)
I20260806 09:11:18.008872 12527 webserver.cc:460] Webserver started at http://127.3.69.3:38769/ using document root <none> and password file <none>
I20260806 09:11:18.009712 12527 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:18.009859 12527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:18.010182 12527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:18.013182 12527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "4e97f231174d4f2f92a35db0df1fad6c"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "40d65929439c0e3b4e192a2454e3aa39"
server_key_iv: "665342c967cac4264764fa65ab50a27a"
server_key_version: "encryptionkey@0"
I20260806 09:11:18.013942 12527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "4e97f231174d4f2f92a35db0df1fad6c"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "40d65929439c0e3b4e192a2454e3aa39"
server_key_iv: "665342c967cac4264764fa65ab50a27a"
server_key_version: "encryptionkey@0"
I20260806 09:11:18.019032 12527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 09:11:18.022619 12545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:18.023948 12527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:18.024135 12527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "4e97f231174d4f2f92a35db0df1fad6c"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "40d65929439c0e3b4e192a2454e3aa39"
server_key_iv: "665342c967cac4264764fa65ab50a27a"
server_key_version: "encryptionkey@0"
I20260806 09:11:18.024360 12527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:18.039906 12527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:18.045189 12527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:18.045501 12527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:18.046557 12527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:18.047923 12527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:18.048032 12527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:18.048182 12527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:18.048269 12527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:18.078143 12335 heartbeater.cc:503] Master 127.3.69.62:46261 was elected leader, sending a full tablet report...
I20260806 09:11:18.080070 12527 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:36007
I20260806 09:11:18.080124 12693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:36007 every 8 connection(s)
I20260806 09:11:18.081408 12527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:18.085844 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 12527
I20260806 09:11:18.086040 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:11:18.086666 3348 external_mini_cluster.cc:1428] Setting key 6afc730369b624116433000e7ec98013
Aug 06 09:11:18 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007477, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:18.102408 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.083988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:46143 (local address 127.3.69.62:46261)
0806 09:11:18.084204 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:18.084211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:18.085428 (+ 1217us) server_negotiation.cc:407] Connection header received
0806 09:11:18.087910 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:18.087913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:18.088064 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:18.088161 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:18.090161 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.090906 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.092496 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.092871 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.092965 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.096440 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:18.096445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:18.096448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:18.096483 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:18.098997 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.099785 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.099789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.099791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.099910 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.100386 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.100391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.100394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.100786 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:18.100933 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:18.101828 (+ 895us) server_negotiation.cc:299] Negotiation successful
0806 09:11:18.102081 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:11:18.103494 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.084516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.3:46143)
0806 09:11:18.085246 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:18.085283 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:18.087595 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:18.088400 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:18.088408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:18.089140 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:18.089970 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.089999 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.091057 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.091061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.092370 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.092383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.093058 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.093063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.093275 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.093907 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:18.093931 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:18.096241 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:18.099206 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.099213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.099224 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.099672 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.100040 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.100044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.100046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.100256 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.100925 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:18.100939 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:18.101367 (+ 428us) client_negotiation.cc:769] Sending connection context
0806 09:11:18.101697 (+ 330us) client_negotiation.cc:240] Negotiation successful
0806 09:11:18.102230 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":287,"threads_started":1}
I20260806 09:11:18.104892 12694 heartbeater.cc:347] Connected to a master server at 127.3.69.62:46261
I20260806 09:11:18.105214 12694 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:18.105957 12694 heartbeater.cc:511] Master 127.3.69.62:46261 requested a full tablet report, sending...
I20260806 09:11:18.107427 12076 ts_manager.cc:194] Registered new tserver with Master: 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007)
I20260806 09:11:18.108240 12076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:46143
I20260806 09:11:18.114676 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.69.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.61.keytab.
Entry for principal oryx/127.3.69.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.61.keytab.
I20260806 09:11:18.168838 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.61:46759
--webserver_interface=127.3.69.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.61.keytab
--principal=oryx/127.3.69.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:46261,127.3.69.61:46759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:18.506439 12701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:18.506790 12701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:18.506963 12701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:18.514098 12701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:18.514248 12701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:18.514350 12701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:18.514454 12701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:18.514540 12701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:18.524452 12701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:46261,127.3.69.61:46759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.61.keytab
--principal=oryx/127.3.69.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.61:46759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.69.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
W20260806 09:11:18.524848 12701 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.69.62:46261,127.3.69.61:46759'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 09:11:18.526175 12701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:18.527884 12701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:18.540226 12715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:18.540267 12712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:18.540313 12711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:18.544628 12701 server_base.cc:1034] running on GCE node
I20260806 09:11:18.545426 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:18.546842 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:18.548102 12701 hybrid_clock.cc:648] HybridClock initialized: now 1786007478548026 us; error 45 us; skew 500 ppm
I20260806 09:11:18.548477 12701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:18 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007478, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:18.552466 12701 init.cc:377] Logged in from keytab as oryx/127.3.69.61@KRBTEST.COM (short username oryx)
I20260806 09:11:18.554663 12701 webserver.cc:460] Webserver started at http://127.3.69.61:36621/ using document root <none> and password file <none>
I20260806 09:11:18.555426 12701 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:18.555518 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:18.555819 12701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:18.558758 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data/instance:
uuid: "a72f9d9f4c654884a17110a56b53d460"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "c30485db864e1bb0cf5d5690050a1d37"
server_key_iv: "8512095e9e8ebc6af8fd785854b1b973"
server_key_version: "encryptionkey@0"
I20260806 09:11:18.559587 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal/instance:
uuid: "a72f9d9f4c654884a17110a56b53d460"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "c30485db864e1bb0cf5d5690050a1d37"
server_key_iv: "8512095e9e8ebc6af8fd785854b1b973"
server_key_version: "encryptionkey@0"
I20260806 09:11:18.564715 12701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:11:18.568032 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:18.569617 12701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:11:18.569774 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
uuid: "a72f9d9f4c654884a17110a56b53d460"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "c30485db864e1bb0cf5d5690050a1d37"
server_key_iv: "8512095e9e8ebc6af8fd785854b1b973"
server_key_version: "encryptionkey@0"
I20260806 09:11:18.570010 12701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:18.585608 12701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:18.589913 12701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:18.590143 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:18.601444 12517 heartbeater.cc:503] Master 127.3.69.62:46261 was elected leader, sending a full tablet report...
I20260806 09:11:18.608845 12701 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.61:46759
I20260806 09:11:18.608878 12806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.61:46759 every 8 connection(s)
I20260806 09:11:18.610167 12701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
I20260806 09:11:18.611696 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 12701
I20260806 09:11:18.614631 12808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:18.621383 12808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:11:18 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.61@KRBTEST.COM
Aug 06 09:11:18 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007478, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.61@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:18.635269 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.613792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.3.69.61:46759)
0806 09:11:18.614580 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:18.614593 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:18.614636 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:11:18.616336 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:18.616357 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:18.616700 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:18.617099 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:18.618654 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.620047 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.621293 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.621977 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.622188 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.627459 (+ 5271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:18.627466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:18.627475 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:18.627507 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:18.631216 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.631855 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.631863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.631870 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.632043 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.632514 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.632518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.632520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.633005 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:18.633291 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:18.633770 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 09:11:18.634142 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":359,"threads_started":1}
I20260806 09:11:18.639914 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.623937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44064)
0806 09:11:18.624852 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:18.624891 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:18.625040 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:18.625528 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:18.625539 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:18.626486 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:18.626866 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.626879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.628176 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.628183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.629580 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.629595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.630575 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.630580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.630704 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.631424 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:18.631441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:18.633968 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:18.637436 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.637445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.637449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.638020 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.638450 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.638453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.638455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.638586 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.639136 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:18.639142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:18.639275 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 09:11:18.639418 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 09:11:18.639645 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":545,"threads_started":1}
I20260806 09:11:18.639940 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.623938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.3.69.62:46261)
0806 09:11:18.624449 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:18.624457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:18.624992 (+ 535us) server_negotiation.cc:407] Connection header received
0806 09:11:18.625182 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:18.625189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:18.625293 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:18.625417 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:18.627174 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.627995 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.629721 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.630362 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.630519 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.634192 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:18.634197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:18.634200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:18.634232 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:18.637302 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.638149 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.638155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.638158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.638341 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.638681 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.638684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.638686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.638969 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:18.639151 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:18.639506 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 09:11:18.639665 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
I20260806 09:11:18.641585 12808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:11:18 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007478, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.61@KRBTEST.COM for oryx/127.3.69.61@KRBTEST.COM
I20260806 09:11:18.653264 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.642163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.0.0.1:58960)
0806 09:11:18.642333 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:18.642351 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:18.642440 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:18.642789 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:18.642793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:18.643158 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:18.643280 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.643288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.644321 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.644326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.645193 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.645205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.645919 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.645924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.646052 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.646580 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:18.646597 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:18.648454 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:18.651216 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.651222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.651227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.651671 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.652032 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.652035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.652037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.652157 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.652607 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:18.652610 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:18.652724 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 09:11:18.652837 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 09:11:18.653039 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 09:11:18.653546 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.642402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.3.69.61:46759)
0806 09:11:18.642528 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:18.642532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:18.642546 (+ 14us) server_negotiation.cc:407] Connection header received
0806 09:11:18.642581 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:18.642585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:18.642631 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:18.642747 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:18.643409 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.644174 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.645347 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.645765 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.645874 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.648620 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:18.648624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:18.648627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:18.648655 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:18.651035 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.651778 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.651782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.651785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.651924 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.652251 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.652255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.652257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.652506 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:18.652616 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:18.652927 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 09:11:18.653166 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":48}
I20260806 09:11:18.657881 12808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Bootstrap starting.
I20260806 09:11:18.660880 12808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:18.661836 12808 log.cc:824] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:18.664379 12808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: No bootstrap required, opened a new log
I20260806 09:11:18.671164 12808 raft_consensus.cc:348] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } }
I20260806 09:11:18.671505 12808 raft_consensus.cc:374] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:18.671591 12808 raft_consensus.cc:729] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72f9d9f4c654884a17110a56b53d460, State: Initialized, Role: FOLLOWER
I20260806 09:11:18.672137 12808 consensus_queue.cc:260] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } }
I20260806 09:11:18.672916 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } } }
I20260806 09:11:18.673254 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:11:18.673857 12808 sys_catalog.cc:564] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 09:11:18.683336 12829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:11:18.683588 12829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:11:18.686918 12701 master_runner.cc:428] Detected that this master a72f9d9f4c654884a17110a56b53d460 is joining an existing cluster
I20260806 09:11:18.687017 12701 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.69.61:46759
I20260806 09:11:18.708541 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.691313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.3.69.62:46261)
0806 09:11:18.691497 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:18.691504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:18.692213 (+ 709us) server_negotiation.cc:407] Connection header received
0806 09:11:18.692495 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:18.692503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:18.692608 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:18.692688 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:18.694440 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.695633 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.697753 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.698400 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.698504 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.701317 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:18.701324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:18.701328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:18.701371 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:18.705119 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.706005 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.706014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.706018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.706199 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.706734 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.706741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.706746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.707192 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:18.707289 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:18.707833 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 09:11:18.708059 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 09:11:18.708786 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.691336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44066)
0806 09:11:18.692109 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:18.692128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:18.692331 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:18.692942 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:18.692947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:18.693830 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:18.694249 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.694263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.695834 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.695841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.697568 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.697583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.698818 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.698825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.699015 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.699818 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:18.699843 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:18.701098 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:18.705261 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.705267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.705272 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.705838 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.706371 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.706377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.706381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.706595 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.707371 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:18.707374 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:18.707534 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 09:11:18.707727 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 09:11:18.708054 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":443,"threads_started":1}
I20260806 09:11:18.725739 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.692473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.0.0.1:58966)
0806 09:11:18.704987 (+ 12514us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:18.705005 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:18.705163 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:18.705966 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:18.705970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:18.706727 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:18.707066 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.707101 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.708611 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.708617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.710374 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.710390 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.712192 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.712202 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.712402 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.714234 (+ 1832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:18.714257 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:18.715307 (+ 1050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:18.722554 (+ 7247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.722562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.722568 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.723127 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.723584 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.723591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.723595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.723758 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.724508 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:18.724512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:18.724705 (+ 193us) client_negotiation.cc:769] Sending connection context
0806 09:11:18.724868 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 09:11:18.725201 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12371,"thread_start_us":12775,"threads_started":1}
I20260806 09:11:18.725769 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.693017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.3.69.61:46759)
0806 09:11:18.693174 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:18.693179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:18.705109 (+ 11930us) server_negotiation.cc:407] Connection header received
0806 09:11:18.705605 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:18.705611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:18.705690 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:18.705797 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:18.707247 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.708384 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.710560 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.711342 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.711450 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.715497 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:18.715503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:18.715508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:18.715556 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:18.720499 (+ 4943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.723268 (+ 2769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.723274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.723277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.723440 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.723884 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.723889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.723892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.724261 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:18.724451 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:18.725008 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 09:11:18.725412 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":40}
WARNING: no policy specified for oryx/127.3.69.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.60@KRBTEST.COM" created.
Aug 06 09:11:18 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007476, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.62@KRBTEST.COM for oryx/127.3.69.61@KRBTEST.COM
I20260806 09:11:18.741900 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.716328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.0.0.1:58976)
0806 09:11:18.723851 (+ 7523us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:18.723913 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:18.724107 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:18.725533 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:18.725540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:18.726517 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:18.726861 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.726875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.728507 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.728516 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.729550 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.729563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.730305 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.730311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.730511 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.731180 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:18.731197 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:18.735978 (+ 4781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:18.739117 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.739128 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.739136 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.739535 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.739957 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.739963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.739967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.740139 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.740725 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:18.740733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:18.740919 (+ 186us) client_negotiation.cc:769] Sending connection context
0806 09:11:18.741107 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 09:11:18.741434 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7208,"thread_start_us":9685,"threads_started":1}
I20260806 09:11:18.741922 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.716377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.3.69.61:46759)
0806 09:11:18.724798 (+ 8421us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:18.724802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:18.724821 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:11:18.724934 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:18.724938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:18.725010 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:18.725120 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:18.727063 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.728370 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.729700 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.730142 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.730278 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.736211 (+ 5933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:18.736216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:18.736220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:18.736256 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:18.738939 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.739684 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.739691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.739695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.739833 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.740265 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.740269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.740271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.740576 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:18.740741 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:18.741216 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 09:11:18.741433 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8305,"thread_start_us":8169,"threads_started":1}
I20260806 09:11:18.744443 12076 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.69.61:46759: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a72f9d9f4c654884a17110a56b53d460"
member_type: NON_VOTER
last_known_addr {
host: "127.3.69.61"
port: 46759
}
attrs {
promote: true
}
}
dest_uuid: "7f4dfd6248ab451497efc55c64e34050"
cas_config_opid_index: -1
I20260806 09:11:18.745769 12076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } }
I20260806 09:11:18.761396 12767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:18.761106 12850 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a72f9d9f4c654884a17110a56b53d460 (127.3.69.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } }
I20260806 09:11:18.765183 12767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Refusing update from remote peer 7f4dfd6248ab451497efc55c64e34050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 09:11:18.772142 12850 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.69.61:46759
I20260806 09:11:18.774287 12701 master.cc:537] Master@127.3.69.61:46759 shutting down...
I20260806 09:11:18.777266 12851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:11:18.780946 12055 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 09:11:18.782603 12055 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 -> Peer a72f9d9f4c654884a17110a56b53d460 (127.3.69.61:46759): Couldn't send request to peer a72f9d9f4c654884a17110a56b53d460. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 09:11:18.776250 12849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } }
I20260806 09:11:18.783548 12849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.60.keytab.
Entry for principal oryx/127.3.69.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.60.keytab.
I20260806 09:11:18.788113 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.60:35943
--webserver_interface=127.3.69.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.60.keytab
--principal=oryx/127.3.69.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:46261,127.3.69.61:46759,127.3.69.60:35943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:11:18.796096 12701 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:18.796626 12701 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:18.796841 12701 tablet_replica.cc:330] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: stopping tablet replica
I20260806 09:11:18.814873 12701 master.cc:552] Master@127.3.69.61:46759 shutdown complete.
I20260806 09:11:18.815050 12701 master_runner.cc:305] Clearing existing system tablet
I20260806 09:11:18.815654 12701 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:18.815773 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:18.819062 12701 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 09:11:18.820773 12861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:18.821190 12701 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:18.821455 12701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 09:11:18.821544 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
uuid: "a72f9d9f4c654884a17110a56b53d460"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "c30485db864e1bb0cf5d5690050a1d37"
server_key_iv: "8512095e9e8ebc6af8fd785854b1b973"
server_key_version: "encryptionkey@0"
I20260806 09:11:18.827188 12701 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:18.834190 12701 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 09:11:18.834372 12701 log.cc:1196] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 09:11:18.834848 12701 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Deleting consensus metadata
I20260806 09:11:18.835213 12701 master_runner.cc:315] Copying system tablet from 127.3.69.62:46261
I20260806 09:11:18.840508 12701 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: tablet copy: Beginning tablet copy session from remote peer at address 127.3.69.62:46261
I20260806 09:11:18.863214 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.841640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.3.69.62:46261)
0806 09:11:18.841834 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:18.841840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:18.847764 (+ 5924us) server_negotiation.cc:407] Connection header received
0806 09:11:18.847972 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:18.847978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:18.848084 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:18.848206 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:18.849490 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.850673 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.852588 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.853294 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.853426 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.856323 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:18.856331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:18.856339 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:18.856384 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:18.860080 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.860909 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.860915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.860919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.861063 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:18.861515 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:18.861521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:18.861524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:18.861896 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:18.862047 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:18.862567 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 09:11:18.862808 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:11:18.863222 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:18.841620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44072)
0806 09:11:18.847597 (+ 5977us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:18.847627 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:18.847814 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:18.848281 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:18.848289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:18.848961 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:18.849307 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.849323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.850840 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.850845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.852412 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:18.852428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:18.853500 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:18.853508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:18.853759 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:18.854626 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:18.854656 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:18.856073 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:18.860238 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.860243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.860247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.860764 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.861190 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:18.861195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:18.861198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:18.861381 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:18.862041 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:18.862046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:18.862245 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 09:11:18.862404 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 09:11:18.862749 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5804,"thread_start_us":5642,"threads_started":1}
I20260806 09:11:18.864777 12105 tablet_copy_service.cc:140] P 7f4dfd6248ab451497efc55c64e34050: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a72f9d9f4c654884a17110a56b53d460 ({username='oryx', principal='oryx/127.3.69.61@KRBTEST.COM'} at 127.0.0.1:44072)
I20260806 09:11:18.865216 12105 tablet_copy_service.cc:161] P 7f4dfd6248ab451497efc55c64e34050: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a72f9d9f4c654884a17110a56b53d460 at {username='oryx', principal='oryx/127.3.69.61@KRBTEST.COM'} at 127.0.0.1:44072: session id = a72f9d9f4c654884a17110a56b53d460-00000000000000000000000000000000
I20260806 09:11:18.870177 12105 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 09:11:18.872982 12701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 09:11:18.880213 12701 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: tablet copy: Starting download of 0 data blocks...
I20260806 09:11:18.880599 12701 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: tablet copy: Starting download of 1 WAL segments...
I20260806 09:11:18.884392 12701 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 09:11:18.888991 12105 tablet_copy_service.cc:342] P 7f4dfd6248ab451497efc55c64e34050: Request end of tablet copy session a72f9d9f4c654884a17110a56b53d460-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.69.61@KRBTEST.COM'} at 127.0.0.1:44072
I20260806 09:11:18.889252 12105 tablet_copy_service.cc:433] P 7f4dfd6248ab451497efc55c64e34050: ending tablet copy session a72f9d9f4c654884a17110a56b53d460-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a72f9d9f4c654884a17110a56b53d460
I20260806 09:11:18.902570 12701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:18.905400 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:18.908407 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:18.919162 12701 server_base.cc:1034] running on GCE node
W20260806 09:11:18.919453 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:18.920015 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:18.920816 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:18.922103 12701 hybrid_clock.cc:648] HybridClock initialized: now 1786007478922032 us; error 35 us; skew 500 ppm
I20260806 09:11:18.922356 12701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:18 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007478, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:18.924612 12701 init.cc:377] Logged in from keytab as oryx/127.3.69.61@KRBTEST.COM (short username oryx)
I20260806 09:11:18.925951 12701 webserver.cc:460] Webserver started at http://127.3.69.61:44429/ using document root <none> and password file <none>
I20260806 09:11:18.926332 12701 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:18.926425 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:18.929611 12701 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260806 09:11:18.931890 12883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:18.932327 12701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 09:11:18.932528 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
uuid: "a72f9d9f4c654884a17110a56b53d460"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "c30485db864e1bb0cf5d5690050a1d37"
server_key_iv: "8512095e9e8ebc6af8fd785854b1b973"
server_key_version: "encryptionkey@0"
I20260806 09:11:18.932821 12701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:18.956238 12701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:18.960456 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:18.990922 12701 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.61:46759
I20260806 09:11:18.991200 12956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.61:46759 every 8 connection(s)
I20260806 09:11:18.992064 12701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
I20260806 09:11:18.997792 12957 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:11:19.002818 12957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Bootstrap starting.
I20260806 09:11:19.027515 12957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:11:19.028182 12957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: Bootstrap complete.
I20260806 09:11:19.029371 12957 raft_consensus.cc:348] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } }
I20260806 09:11:19.029711 12957 raft_consensus.cc:729] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a72f9d9f4c654884a17110a56b53d460, State: Initialized, Role: LEARNER
I20260806 09:11:19.030002 12957 consensus_queue.cc:260] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } }
I20260806 09:11:19.036861 12962 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } }
I20260806 09:11:19.037207 12962 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:11:19.037951 12957 sys_catalog.cc:564] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:19.051827 12973 catalog_manager.cc:1259] Loaded cluster ID: e1c2b4515b97401c906286923abfccca
I20260806 09:11:19.052000 12973 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: loading cluster ID for follower catalog manager: success
I20260806 09:11:19.055826 12973 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: acquiring CA information for follower catalog manager: success
I20260806 09:11:19.058583 12973 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:11:19.110940 12694 heartbeater.cc:503] Master 127.3.69.62:46261 was elected leader, sending a full tablet report...
W20260806 09:11:19.201251 12859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:19.201610 12859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:19.201725 12859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:19.209136 12859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:19.209229 12859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:19.209285 12859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:19.209342 12859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:19.209388 12859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:19.219208 12859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:46261,127.3.69.61:46759,127.3.69.60:35943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.60.keytab
--principal=oryx/127.3.69.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.60:35943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.69.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:19.220438 12859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:19.221719 12859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:19.234470 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:19.234493 12981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:19.234669 12979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:19.235420 12859 server_base.cc:1034] running on GCE node
I20260806 09:11:19.236227 12859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:19.237668 12859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:19.238894 12859 hybrid_clock.cc:648] HybridClock initialized: now 1786007479238855 us; error 33 us; skew 500 ppm
I20260806 09:11:19.239307 12859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:19 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007479, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:19.243358 12859 init.cc:377] Logged in from keytab as oryx/127.3.69.60@KRBTEST.COM (short username oryx)
I20260806 09:11:19.245191 12859 webserver.cc:460] Webserver started at http://127.3.69.60:43107/ using document root <none> and password file <none>
I20260806 09:11:19.246019 12859 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:19.246199 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:19.246526 12859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:19.248112 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.240363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.0.0.1:58986)
0806 09:11:19.240658 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.240677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.240828 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.241558 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.241562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.241591 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:19.241740 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.241748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.243621 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.243630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.245939 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.245954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.247279 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.247286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.247497 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.247572 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.247689 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.247837 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260806 09:11:19.248137 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.240548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.3.69.61:46759)
0806 09:11:19.241172 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.241178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.241203 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:11:19.241279 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.241285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.241386 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.241514 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:19.241898 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.243381 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.246094 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.247051 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.247222 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.247392 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.247763 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.247894 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":323,"threads_started":1}
I20260806 09:11:19.249363 12927 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 09:11:19.250190 12962 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f4dfd6248ab451497efc55c64e34050. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } }
I20260806 09:11:19.250461 12962 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:11:19.250524 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data/instance:
uuid: "fc23bdd3625d48268b028d395e9659b7"
format_stamp: "Formatted at 2026-08-06 09:11:19 on dist-test-slave-w4v5"
server_key: "687b6e84d386b7fa13b61f71d1c56fb9"
server_key_iv: "b06c2e1458775719c0430ae758492b6a"
server_key_version: "encryptionkey@0"
I20260806 09:11:19.251358 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal/instance:
uuid: "fc23bdd3625d48268b028d395e9659b7"
format_stamp: "Formatted at 2026-08-06 09:11:19 on dist-test-slave-w4v5"
server_key: "687b6e84d386b7fa13b61f71d1c56fb9"
server_key_iv: "b06c2e1458775719c0430ae758492b6a"
server_key_version: "encryptionkey@0"
I20260806 09:11:19.256505 12859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:11:19.259971 12995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:19.261390 12859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:11:19.261615 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
uuid: "fc23bdd3625d48268b028d395e9659b7"
format_stamp: "Formatted at 2026-08-06 09:11:19 on dist-test-slave-w4v5"
server_key: "687b6e84d386b7fa13b61f71d1c56fb9"
server_key_iv: "b06c2e1458775719c0430ae758492b6a"
server_key_version: "encryptionkey@0"
I20260806 09:11:19.261817 12859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:19.280702 12859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:19.285203 12859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:19.285493 12859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:19.305971 12859 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.60:35943
I20260806 09:11:19.306015 13066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.60:35943 every 8 connection(s)
I20260806 09:11:19.307490 12859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
I20260806 09:11:19.311441 13068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:19.317272 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 12859
I20260806 09:11:19.317914 13068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:11:19 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007479, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.60@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
Aug 06 09:11:19 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.60@KRBTEST.COM
I20260806 09:11:19.338429 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.320891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.3.69.62:46261)
0806 09:11:19.321082 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.321086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.321885 (+ 799us) server_negotiation.cc:407] Connection header received
0806 09:11:19.322870 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.322878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.322965 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.323140 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.324427 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.325149 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.327017 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.327531 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.327675 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.331619 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.331625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.331630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.331673 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.334948 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.335931 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.335938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.335941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.336125 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.336593 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.336600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.336607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.336867 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.337059 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.337744 (+ 685us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.337996 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:11:19.339771 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.320916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44086)
0806 09:11:19.321729 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.321763 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.322589 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.323179 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.323186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.323837 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:19.324253 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.324270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.325316 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.325320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.326788 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.326824 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.327781 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.327791 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.327982 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.328789 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:19.328824 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:19.331443 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:19.335115 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.335126 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.335137 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.335769 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.336259 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.336262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.336265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.336457 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.337001 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:19.337009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:19.337348 (+ 339us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.337620 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.338117 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":90,"thread_start_us":454,"threads_started":1}
I20260806 09:11:19.339790 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.319403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.3.69.60:35943)
0806 09:11:19.320142 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.320153 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.320187 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:11:19.322577 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.322596 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.322953 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.323200 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.324879 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.326138 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.327443 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.328054 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.328205 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.333059 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.333068 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.333080 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.333113 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.336359 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.337058 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.337065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.337067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.337193 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.337594 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.337597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.337599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.338029 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.338335 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.338674 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.338929 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":334,"threads_started":1}
I20260806 09:11:19.341293 13068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:11:19 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007479, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.60@KRBTEST.COM for oryx/127.3.69.61@KRBTEST.COM
I20260806 09:11:19.353868 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.341946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.0.0.1:59000)
0806 09:11:19.342144 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.342163 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.342260 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.342625 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.342629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.343092 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:19.343251 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.343259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.344457 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.344464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.345457 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.345468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.346205 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.346210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.346411 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.346936 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:19.346953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:19.348918 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:19.351972 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.351975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.351978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.352346 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.352668 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.352671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.352673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.352794 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.353240 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:19.353242 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:19.353354 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.353459 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.353652 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 09:11:19.353981 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.342137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.3.69.61:46759)
0806 09:11:19.342328 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.342335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.342354 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:11:19.342413 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.342416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.342477 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.342584 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.343390 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.344337 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.345631 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.346048 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.346179 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.349106 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.349110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.349114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.349146 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.351853 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.352444 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.352448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.352450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.352574 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.352885 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.352889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.352891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.353143 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.353261 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.353546 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.353714 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:11:19.354892 13068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:11:19 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007479, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.60@KRBTEST.COM for oryx/127.3.69.60@KRBTEST.COM
I20260806 09:11:19.367621 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.355807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.3.69.60:35943)
0806 09:11:19.356013 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.356019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.356119 (+ 100us) server_negotiation.cc:407] Connection header received
0806 09:11:19.356303 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.356306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.356356 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.356452 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.357104 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.357762 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.358859 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.359330 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.359467 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.362307 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.362312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.362316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.362341 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.364893 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.365537 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.365544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.365547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.365656 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.366068 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.366073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.366076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.366454 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.366617 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.366942 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.367299 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":224,"server-negotiator.queue_time_us":83}
I20260806 09:11:19.367594 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.355816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.0.0.1:60648)
0806 09:11:19.356015 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.356035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.356179 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.356469 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.356473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.356838 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:19.356963 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.356979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.357890 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.357897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.358715 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.358726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.359475 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.359480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.359718 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.360162 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:19.360178 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:19.362125 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:19.365024 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.365038 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.365042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.365392 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.365778 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.365783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.365789 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.365942 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.366581 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:19.366586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:19.366718 (+ 132us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.366859 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.367180 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:11:19.372581 13068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Bootstrap starting.
I20260806 09:11:19.375725 13068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:19.376755 13068 log.cc:824] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:19.379509 13068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: No bootstrap required, opened a new log
I20260806 09:11:19.386597 13068 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } }
I20260806 09:11:19.386905 13068 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:19.386993 13068 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc23bdd3625d48268b028d395e9659b7, State: Initialized, Role: FOLLOWER
I20260806 09:11:19.387598 13068 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } }
I20260806 09:11:19.388437 13075 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } } }
I20260806 09:11:19.388803 13075 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:11:19.389391 13068 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 09:11:19.399940 13091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:11:19.400103 13091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:11:19.403736 12859 master_runner.cc:428] Detected that this master fc23bdd3625d48268b028d395e9659b7 is joining an existing cluster
I20260806 09:11:19.403882 12859 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.69.60:35943
I20260806 09:11:19.430318 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.408963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.3.69.61:46759)
0806 09:11:19.409164 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.409170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.410822 (+ 1652us) server_negotiation.cc:407] Connection header received
0806 09:11:19.411182 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.411192 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.411312 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.411405 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.413372 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.414830 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.417537 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.418209 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.418338 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.421722 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.421731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.421737 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.421783 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.425636 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.426522 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.426528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.426534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.426699 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.427256 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.427262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.427265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.428820 (+ 1555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.428965 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.429678 (+ 713us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.429944 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:11:19.431504 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.409656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.0.0.1:59016)
0806 09:11:19.410550 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.410585 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.410948 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.411688 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.411695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.412665 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:19.413139 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.413160 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.415060 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.415068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.417355 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.417376 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.418602 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.418607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.418854 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.419890 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:19.419913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:19.421529 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:19.425792 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.425798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.425806 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.426378 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.426846 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.426851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.426858 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.427056 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.429067 (+ 2011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:19.429074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:19.429347 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.430602 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.430968 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":28,"thread_start_us":556,"threads_started":1}
I20260806 09:11:19.436342 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.409703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.0.0.1:60658)
0806 09:11:19.418308 (+ 8605us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.418331 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.418476 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.418996 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.418999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.420098 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:19.421886 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.421900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.423511 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.423516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.425179 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.425194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.426577 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.426585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.426762 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.427430 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:19.427533 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:19.428800 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:19.433218 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.433223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.433227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.433719 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.434216 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.434220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.434224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.434398 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.435186 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:19.435196 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:19.435357 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.435496 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.435833 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8440,"thread_start_us":10790,"threads_started":1}
I20260806 09:11:19.436411 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.409331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.3.69.60:35943)
0806 09:11:19.409593 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.409599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.418395 (+ 8796us) server_negotiation.cc:407] Connection header received
0806 09:11:19.418597 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.418602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.418672 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.418748 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.422056 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.423295 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.425327 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.426132 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.426228 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.428982 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.428988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.428993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.429027 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.432850 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.433850 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.433856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.433859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.434081 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.434506 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.434511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.434514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.434940 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.435120 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.435603 (+ 483us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.435985 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":197,"server-negotiator.queue_time_us":71}
I20260806 09:11:19.437310 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.408924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.3.69.62:46261)
0806 09:11:19.409117 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.409123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.421083 (+ 11960us) server_negotiation.cc:407] Connection header received
0806 09:11:19.421275 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.421281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.421360 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.421470 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.423296 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.424446 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.426671 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.427361 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.427455 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.430231 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.430237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.430241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.430280 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.434411 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.435186 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.435192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.435197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.435343 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.435786 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.435792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.435795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.436164 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.436304 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.436906 (+ 602us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.437068 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 09:11:19.438196 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.410818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44094)
0806 09:11:19.420956 (+ 10138us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.420973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.421129 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.421496 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.421499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.422636 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:19.423049 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.423063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.424682 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.424687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.426490 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.426504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.427778 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.427785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.428074 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.428693 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:19.428714 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:19.430017 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:19.434602 (+ 4585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.434608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.434613 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.435010 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.435460 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.435465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.435469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.435640 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.436372 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:19.436378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:19.436590 (+ 212us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.437580 (+ 990us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.437805 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10024,"thread_start_us":6579,"threads_started":1}
I20260806 09:11:19.446118 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 12166
Aug 06 09:11:19 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007476, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.62@KRBTEST.COM for oryx/127.3.69.60@KRBTEST.COM
I20260806 09:11:19.459211 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.443574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.0.0.1:60664)
0806 09:11:19.443788 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.443810 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.443930 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.444417 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.444424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.444977 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:19.445134 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.445143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.446469 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.446474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.448009 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.448024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.449167 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.449174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.449357 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.450037 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:19.450065 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:19.452388 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:19.456184 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.456190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.456194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.456778 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.457210 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.457215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.457219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.457386 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.458111 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:19.458116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:19.458278 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.458464 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.458779 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 09:11:19.459235 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.443619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.3.69.60:35943)
0806 09:11:19.443822 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.443826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.443859 (+ 33us) server_negotiation.cc:407] Connection header received
0806 09:11:19.444145 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.444151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.444259 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.444357 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.445283 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.446306 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.448184 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.448980 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.449655 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.452577 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.452587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.452591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.452648 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.456041 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.456922 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.456929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.456932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.457071 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.457513 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.457521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.457526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.457959 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.458126 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.458608 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.458838 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
Aug 06 09:11:19 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.1@KRBTEST.COM
I20260806 09:11:19.461239 12076 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.69.60:35943: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fc23bdd3625d48268b028d395e9659b7"
member_type: NON_VOTER
last_known_addr {
host: "127.3.69.60"
port: 35943
}
attrs {
promote: true
}
}
dest_uuid: "7f4dfd6248ab451497efc55c64e34050"
cas_config_opid_index: 5
I20260806 09:11:19.462325 12076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } }
I20260806 09:11:19.464179 12849 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc23bdd3625d48268b028d395e9659b7 (127.3.69.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.464798 13032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:19.468681 13032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Refusing update from remote peer 7f4dfd6248ab451497efc55c64e34050: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 09:11:19.469149 12849 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.69.60:35943
I20260806 09:11:19.469785 12859 master.cc:537] Master@127.3.69.60:35943 shutting down...
I20260806 09:11:19.470384 12927 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 LEARNER]: Refusing update from remote peer 7f4dfd6248ab451497efc55c64e34050: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 09:11:19.473774 13102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 09:11:19.474469 13102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:19.481989 12859 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:19.482498 12859 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:19.482748 12859 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: stopping tablet replica
I20260806 09:11:19.484056 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.447824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.3.69.1:45171)
0806 09:11:19.448452 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.448461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.448489 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:11:19.448554 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.448562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.448794 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.449066 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.450760 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.452454 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.454776 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.455727 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.455865 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.461177 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.461192 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.461211 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.461259 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.475970 (+ 14711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.477061 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.477071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.477074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.477252 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.478369 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.478374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.478378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.481707 (+ 3329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.482026 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.483342 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.483623 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":570,"threads_started":1}
I20260806 09:11:19.485459 12330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:19.491837 13103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.492380 13103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:19.493690 12859 master.cc:552] Master@127.3.69.60:35943 shutdown complete.
I20260806 09:11:19.493770 12859 master_runner.cc:305] Clearing existing system tablet
I20260806 09:11:19.494160 12859 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:19.494282 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:19.497198 12927 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fc23bdd3625d48268b028d395e9659b7 (127.3.69.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.501844 12962 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: NON_VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: true } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.502159 12962 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:11:19.503284 13103 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: attempting to promote NON_VOTER a72f9d9f4c654884a17110a56b53d460 to VOTER
I20260806 09:11:19.503512 12859 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.005s
I20260806 09:11:19.504148 13103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } }
I20260806 09:11:19.506340 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.505678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.0.0.1:60672)
0806 09:11:19.505987 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.506132 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260806 09:11:19.507043 12926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 LEARNER]: Refusing update from remote peer 7f4dfd6248ab451497efc55c64e34050: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 09:11:19.507865 12849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 09:11:19.509375 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.507756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.0.0.1:60688)
0806 09:11:19.509049 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.509158 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1151}
W20260806 09:11:19.510735 12056 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 -> Peer fc23bdd3625d48268b028d395e9659b7 (127.3.69.60:35943): Couldn't send request to peer fc23bdd3625d48268b028d395e9659b7. Status: Network error: Client connection negotiation failed: client connection to 127.3.69.60:35943: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 09:11:19.512290 12849 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a72f9d9f4c654884a17110a56b53d460 (127.3.69.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.513504 13110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:19.513849 12859 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:19.514119 12859 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260806 09:11:19.514237 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
uuid: "fc23bdd3625d48268b028d395e9659b7"
format_stamp: "Formatted at 2026-08-06 09:11:19 on dist-test-slave-w4v5"
server_key: "687b6e84d386b7fa13b61f71d1c56fb9"
server_key_iv: "b06c2e1458775719c0430ae758492b6a"
server_key_version: "encryptionkey@0"
I20260806 09:11:19.515717 12926 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a72f9d9f4c654884a17110a56b53d460 (127.3.69.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.519917 12859 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:19.521270 13102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.521545 13102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:19.521842 13102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.522046 13102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:19.522809 12962 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.523052 12962 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:11:19.525838 12859 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 09:11:19.525959 12859 log.cc:1196] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 09:11:19.526316 12859 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Deleting consensus metadata
I20260806 09:11:19.526512 12859 master_runner.cc:315] Copying system tablet from 127.3.69.62:46261
I20260806 09:11:19.532867 12859 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: tablet copy: Beginning tablet copy session from remote peer at address 127.3.69.62:46261
I20260806 09:11:19.547832 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.533584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44100)
0806 09:11:19.535488 (+ 1904us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:19.535511 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:19.535681 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:19.536068 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:19.536074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:19.536580 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:19.536800 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.536809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.537895 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.537901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.538981 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:19.538994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.539918 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.539924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:19.540174 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.540858 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:19.540876 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:19.541584 (+ 708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:19.545201 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.545206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.545214 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.545572 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.545952 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:19.545957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:19.545961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:19.546111 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:19.546631 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:19.546636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:19.546809 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 09:11:19.546952 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 09:11:19.547339 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1768,"thread_start_us":1665,"threads_started":1}
I20260806 09:11:19.548549 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:19.533584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.3.69.62:46261)
0806 09:11:19.533832 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:19.533841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:19.535604 (+ 1763us) server_negotiation.cc:407] Connection header received
0806 09:11:19.535821 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:19.535827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:19.535909 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:19.536016 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:19.536950 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.537755 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.539164 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:19.539741 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:19.539881 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:19.541788 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:19.541794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:19.541799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:19.541843 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:19.545057 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.545696 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.545703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.545706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.545837 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:19.546232 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:19.546238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:19.546242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:19.546506 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:19.546672 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:19.547060 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 09:11:19.548187 (+ 1127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260806 09:11:19.549065 12105 tablet_copy_service.cc:140] P 7f4dfd6248ab451497efc55c64e34050: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fc23bdd3625d48268b028d395e9659b7 ({username='oryx', principal='oryx/127.3.69.60@KRBTEST.COM'} at 127.0.0.1:44100)
I20260806 09:11:19.549247 12105 tablet_copy_service.cc:161] P 7f4dfd6248ab451497efc55c64e34050: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fc23bdd3625d48268b028d395e9659b7 at {username='oryx', principal='oryx/127.3.69.60@KRBTEST.COM'} at 127.0.0.1:44100: session id = fc23bdd3625d48268b028d395e9659b7-00000000000000000000000000000000
I20260806 09:11:19.551721 12105 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 09:11:19.553331 12859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 09:11:19.559513 12859 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: tablet copy: Starting download of 0 data blocks...
I20260806 09:11:19.559810 12859 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: tablet copy: Starting download of 1 WAL segments...
I20260806 09:11:19.562094 12859 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 09:11:19.565497 12105 tablet_copy_service.cc:342] P 7f4dfd6248ab451497efc55c64e34050: Request end of tablet copy session fc23bdd3625d48268b028d395e9659b7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.69.60@KRBTEST.COM'} at 127.0.0.1:44100
I20260806 09:11:19.565654 12105 tablet_copy_service.cc:433] P 7f4dfd6248ab451497efc55c64e34050: ending tablet copy session fc23bdd3625d48268b028d395e9659b7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fc23bdd3625d48268b028d395e9659b7
I20260806 09:11:19.579437 12859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:19.582002 13118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:19.591984 13121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:19.595233 13119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:19.595752 12859 server_base.cc:1034] running on GCE node
I20260806 09:11:19.596124 12859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:19.596822 12859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:19.598102 12859 hybrid_clock.cc:648] HybridClock initialized: now 1786007479598053 us; error 42 us; skew 500 ppm
I20260806 09:11:19.598343 12859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:19 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007479, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:19.600589 12859 init.cc:377] Logged in from keytab as oryx/127.3.69.60@KRBTEST.COM (short username oryx)
I20260806 09:11:19.601949 12859 webserver.cc:460] Webserver started at http://127.3.69.60:46047/ using document root <none> and password file <none>
I20260806 09:11:19.602362 12859 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:19.602469 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:19.605610 12859 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260806 09:11:19.607750 13133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:19.608211 12859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260806 09:11:19.608356 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
uuid: "fc23bdd3625d48268b028d395e9659b7"
format_stamp: "Formatted at 2026-08-06 09:11:19 on dist-test-slave-w4v5"
server_key: "687b6e84d386b7fa13b61f71d1c56fb9"
server_key_iv: "b06c2e1458775719c0430ae758492b6a"
server_key_version: "encryptionkey@0"
I20260806 09:11:19.608556 12859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:19.623965 12859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:19.628175 12859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:19.641852 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 12166
I20260806 09:11:19.680090 12859 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.60:35943
I20260806 09:11:19.680161 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.60:35943 every 8 connection(s)
I20260806 09:11:19.681324 12859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
I20260806 09:11:19.681381 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:45171
--local_ip_for_outbound_sockets=127.3.69.1
--tserver_master_addrs=127.3.69.62:46261
--webserver_port=35805
--webserver_interface=127.3.69.1
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab
--principal=oryx/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759,127.3.69.60:35943 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:11:19.686820 13212 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:11:19.690697 13212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Bootstrap starting.
I20260806 09:11:19.710203 13212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:11:19.710721 13212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Bootstrap complete.
I20260806 09:11:19.711735 13212 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } }
I20260806 09:11:19.712016 13212 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc23bdd3625d48268b028d395e9659b7, State: Initialized, Role: LEARNER
I20260806 09:11:19.712256 13212 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } }
I20260806 09:11:19.719566 13217 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:19.719873 13212 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:19.720074 13217 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:11:19.728801 13232 catalog_manager.cc:1259] Loaded cluster ID: e1c2b4515b97401c906286923abfccca
I20260806 09:11:19.728896 13232 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: loading cluster ID for follower catalog manager: success
I20260806 09:11:19.732574 13232 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: acquiring CA information for follower catalog manager: success
I20260806 09:11:19.735411 13232 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 09:11:20.036000 13214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:20.036329 13214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:20.036504 13214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:20.044385 13214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:20.044531 13214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:20.044711 13214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:20.054867 13214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab
--principal=oryx/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:45171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=35805
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759,127.3.69.60:35943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:20.056293 13214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:20.057758 13214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:20.071465 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:20.071552 13237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:20.071503 13238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:20.072280 13214 server_base.cc:1034] running on GCE node
I20260806 09:11:20.072919 13214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:20.073952 13214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:20.075189 13214 hybrid_clock.cc:648] HybridClock initialized: now 1786007480075139 us; error 60 us; skew 500 ppm
I20260806 09:11:20.075513 13214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007480, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:20.078744 13214 init.cc:377] Logged in from keytab as oryx/127.3.69.1@KRBTEST.COM (short username oryx)
I20260806 09:11:20.080497 13214 webserver.cc:460] Webserver started at http://127.3.69.1:35805/ using document root <none> and password file <none>
I20260806 09:11:20.081344 13214 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:20.081492 13214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:20.087464 13214 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:11:20.090153 13252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:20.091630 13214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:20.091773 13214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "c59aeb9db55843f8af2e0a3197e5ea96"
format_stamp: "Formatted at 2026-08-06 09:11:16 on dist-test-slave-w4v5"
server_key: "ffe7ad70fe4f957d14d3d23819e272c0"
server_key_iv: "a1424b879b15e61ec13d78910e8cc4dd"
server_key_version: "encryptionkey@0"
I20260806 09:11:20.092782 13214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:20.104040 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.097344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.3.69.60:35943)
0806 09:11:20.098076 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.098081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.098096 (+ 15us) server_negotiation.cc:407] Connection header received
0806 09:11:20.098182 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.098186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.098300 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.098448 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:20.099031 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.100072 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.102174 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.103135 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.103257 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.103400 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.103674 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.103773 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":411,"threads_started":1}
I20260806 09:11:20.104142 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.097155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.0.0.1:60694)
0806 09:11:20.097724 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:20.097748 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:20.097904 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:20.098513 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:20.098519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:20.098567 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:20.098833 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.098850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.100236 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.100242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.102005 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.102020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.103312 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.103318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.103514 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.103547 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:11:20.103643 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 09:11:20.103793 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":354,"threads_started":1}
I20260806 09:11:20.105242 13173 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 09:11:20.105901 13217 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f4dfd6248ab451497efc55c64e34050. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:20.106257 13217 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:11:20.108001 13214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:20.112574 13214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:20.112821 13214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:20.113842 13214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:20.115489 13214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:20.115578 13214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:20.115725 13214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:20.115823 13214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:20.152295 13214 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:45171
I20260806 09:11:20.152388 13416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:45171 every 8 connection(s)
I20260806 09:11:20.153766 13214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:20.159803 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 13214
I20260806 09:11:20.160158 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 12345
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007480, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.61@KRBTEST.COM
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007480, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.60@KRBTEST.COM
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007480, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.2@KRBTEST.COM
I20260806 09:11:20.191546 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.156754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:34277 (local address 127.3.69.61:46759)
0806 09:11:20.157622 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.157628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.158797 (+ 1169us) server_negotiation.cc:407] Connection header received
0806 09:11:20.169765 (+ 10968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.169775 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.169916 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.170024 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:20.173206 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.174292 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.176939 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.177593 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.177697 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.183641 (+ 5944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:20.183648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:20.183653 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:20.183705 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:20.187454 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.188435 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.188442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.188447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.188594 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.189187 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.189192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.189195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.189552 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:20.189684 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.190848 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.191125 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":589,"threads_started":1}
I20260806 09:11:20.191568 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.157045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:33697 (local address 127.3.69.60:35943)
0806 09:11:20.157238 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.157244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.170858 (+ 13614us) server_negotiation.cc:407] Connection header received
0806 09:11:20.171108 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.171114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.171190 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.171263 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:20.174183 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.175668 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.177768 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.178382 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.178494 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.183176 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:20.183182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:20.183186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:20.183227 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:20.187034 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.188390 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.188397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.188401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.188565 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.189165 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.189172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.189176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.189560 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:20.189691 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.190840 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.191124 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:11:20.192081 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.157719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:52179 (local address 127.3.69.62:46261)
0806 09:11:20.158232 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.158240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.170168 (+ 11928us) server_negotiation.cc:407] Connection header received
0806 09:11:20.171063 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.171068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.171181 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.171264 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:20.173857 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.175006 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.176997 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.177624 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.177725 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.184432 (+ 6707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:20.184442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:20.184447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:20.184493 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:20.188848 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.189665 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.189670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.189673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.189836 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.190391 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.190396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.190399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.190805 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:20.190920 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.191472 (+ 552us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.191719 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":253,"threads_started":1}
I20260806 09:11:20.194032 13428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.157448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.3.69.1:34277)
0806 09:11:20.158640 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:20.158677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:20.167952 (+ 9275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:20.170879 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:20.170896 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:20.171945 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:20.172969 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.172997 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.174504 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.174511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.176761 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.176776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.177898 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.177904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.178182 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.178986 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:20.179029 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:20.183420 (+ 4391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:20.187624 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.187631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.187635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.188289 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.188762 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.188769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.188773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.189051 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.189793 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:20.189802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:20.190344 (+ 542us) client_negotiation.cc:769] Sending connection context
0806 09:11:20.191862 (+ 1518us) client_negotiation.cc:240] Negotiation successful
0806 09:11:20.192541 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":38,"spinlock_wait_cycles":22016,"thread_start_us":13021,"threads_started":1}
I20260806 09:11:20.194097 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.157448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.3.69.1:33697)
0806 09:11:20.170729 (+ 13281us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:20.170757 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:20.170948 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:20.171484 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:20.171489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:20.172453 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:20.173986 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.173999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.175916 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.175925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.177600 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.177614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.178691 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.178702 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.178826 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.179389 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:20.179412 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:20.182935 (+ 3523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:20.187364 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.187373 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.187391 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.188240 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.188737 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.188743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.188747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.188978 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.189796 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:20.189808 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:20.190345 (+ 537us) client_negotiation.cc:769] Sending connection context
0806 09:11:20.192388 (+ 2043us) client_negotiation.cc:240] Negotiation successful
0806 09:11:20.192990 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13128,"mutex_wait_us":803,"spinlock_wait_cycles":18048,"thread_start_us":396,"threads_started":1}
I20260806 09:11:20.194779 13426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.157706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.1:52179)
0806 09:11:20.169795 (+ 12089us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:20.169821 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:20.170876 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:20.171487 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:20.171491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:20.173287 (+ 1796us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:20.173642 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.173658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.175219 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.175224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.176849 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.176864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.178012 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.178017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.178177 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.179154 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:20.179171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:20.184169 (+ 4998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:20.189015 (+ 4846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.189019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.189030 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.189523 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.190023 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.190027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.190031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.190243 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.191018 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:20.191022 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:20.191174 (+ 152us) client_negotiation.cc:769] Sending connection context
0806 09:11:20.192340 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0806 09:11:20.192665 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11916,"mutex_wait_us":176,"thread_start_us":12671,"threads_started":1}
I20260806 09:11:20.196153 13420 heartbeater.cc:347] Connected to a master server at 127.3.69.61:46759
I20260806 09:11:20.196187 13417 heartbeater.cc:347] Connected to a master server at 127.3.69.60:35943
I20260806 09:11:20.196242 13419 heartbeater.cc:347] Connected to a master server at 127.3.69.62:46261
I20260806 09:11:20.196540 13419 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:20.197412 13419 heartbeater.cc:511] Master 127.3.69.62:46261 requested a full tablet report, sending...
I20260806 09:11:20.196544 13420 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:20.196550 13417 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:20.198805 13420 heartbeater.cc:511] Master 127.3.69.61:46759 requested a full tablet report, sending...
I20260806 09:11:20.199036 13417 heartbeater.cc:511] Master 127.3.69.60:35943 requested a full tablet report, sending...
I20260806 09:11:20.199328 12076 ts_manager.cc:194] Re-registered known tserver with Master: c59aeb9db55843f8af2e0a3197e5ea96 (127.3.69.1:45171)
I20260806 09:11:20.200459 12076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:52179
I20260806 09:11:20.200637 12911 ts_manager.cc:194] Registered new tserver with Master: c59aeb9db55843f8af2e0a3197e5ea96 (127.3.69.1:45171)
I20260806 09:11:20.200850 13158 ts_manager.cc:194] Registered new tserver with Master: c59aeb9db55843f8af2e0a3197e5ea96 (127.3.69.1:45171)
I20260806 09:11:20.201049 13430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.161175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.3.69.2:40105)
0806 09:11:20.176806 (+ 15631us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.176815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.176843 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:11:20.176908 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.176915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.177148 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.177362 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:20.181201 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.182948 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.185019 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.185928 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.186065 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.192440 (+ 6375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:20.192456 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:20.192473 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:20.192533 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:20.196740 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.197578 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.197587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.197590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.197759 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.198125 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.198130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.198133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.198584 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:20.198894 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.200362 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.200645 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15451,"thread_start_us":16328,"threads_started":1}
I20260806 09:11:20.202356 12508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:20.287545 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 12345
I20260806 09:11:20.311714 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:40105
--local_ip_for_outbound_sockets=127.3.69.2
--tserver_master_addrs=127.3.69.62:46261
--webserver_port=45553
--webserver_interface=127.3.69.2
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab
--principal=oryx/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759,127.3.69.60:35943 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:11:20.491879 13254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: NON_VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: true } } }
I20260806 09:11:20.492431 13254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:20.494045 13254 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: attempting to promote NON_VOTER fc23bdd3625d48268b028d395e9659b7 to VOTER
I20260806 09:11:20.494632 13254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } }
I20260806 09:11:20.496430 13173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 LEARNER]: Refusing update from remote peer 7f4dfd6248ab451497efc55c64e34050: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 09:11:20.496512 12926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Refusing update from remote peer 7f4dfd6248ab451497efc55c64e34050: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 09:11:20.497076 13436 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 09:11:20.497634 13438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 09:11:20.500629 13254 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fc23bdd3625d48268b028d395e9659b7 (127.3.69.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:20.501226 12926 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fc23bdd3625d48268b028d395e9659b7 (127.3.69.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:20.502803 13173 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fc23bdd3625d48268b028d395e9659b7 (127.3.69.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:20.505260 13254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:20.505528 13254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:20.505795 13254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:20.506034 13254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:20.506278 13254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:20.506493 13254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:20.508252 13217 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:20.508544 13217 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:11:20.513891 13446 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f4dfd6248ab451497efc55c64e34050" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:20.514436 13446 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 09:11:20.659968 13434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:20.660243 13434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:20.660338 13434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:20.667497 13434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:20.667599 13434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:20.667711 13434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:20.677336 13434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab
--principal=oryx/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:40105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=45553
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759,127.3.69.60:35943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:20.678591 13434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:20.679926 13434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:20.693497 13455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:20.693557 13456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:20.693755 13458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:20.694582 13434 server_base.cc:1034] running on GCE node
I20260806 09:11:20.695358 13434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:20.696348 13434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:20.697582 13434 hybrid_clock.cc:648] HybridClock initialized: now 1786007480697538 us; error 58 us; skew 500 ppm
I20260806 09:11:20.697917 13434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007480, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:20.701294 13434 init.cc:377] Logged in from keytab as oryx/127.3.69.2@KRBTEST.COM (short username oryx)
I20260806 09:11:20.702812 13434 webserver.cc:460] Webserver started at http://127.3.69.2:45553/ using document root <none> and password file <none>
I20260806 09:11:20.703583 13434 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:20.703747 13434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:20.709028 13434 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 09:11:20.711723 13466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:20.712960 13434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:20.713166 13434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "e4d35582e97a4bfea3165b8f33a3606b"
format_stamp: "Formatted at 2026-08-06 09:11:17 on dist-test-slave-w4v5"
server_key: "f20f7a9705dc60f4b227f6e30c552931"
server_key_iv: "9775b96ffb739921b97397a5cc7855d3"
server_key_version: "encryptionkey@0"
I20260806 09:11:20.714147 13434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:20.730015 13434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:20.734264 13434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:20.734562 13434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:20.735534 13434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:20.736866 13434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:20.737001 13434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:20.737128 13434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:20.737218 13434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:20.782850 13434 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:40105
I20260806 09:11:20.782946 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:40105 every 8 connection(s)
I20260806 09:11:20.784510 13434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:20.786991 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 13434
I20260806 09:11:20.787429 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 12527
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.3@KRBTEST.COM
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007480, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.60@KRBTEST.COM
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007480, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.61@KRBTEST.COM
Aug 06 09:11:20 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007480, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:20.827905 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.789543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:36317 (local address 127.3.69.61:46759)
0806 09:11:20.798585 (+ 9042us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.798591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.808705 (+ 10114us) server_negotiation.cc:407] Connection header received
0806 09:11:20.808966 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.808974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.809079 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.809316 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:20.810743 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.811900 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.814213 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.814855 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.814944 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.819946 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:20.819956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:20.819962 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:20.820023 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:20.824794 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.825625 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.825631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.825634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.825762 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.826276 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.826280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.826283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.826622 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:20.826743 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.827327 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.827560 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8916,"thread_start_us":8869,"threads_started":1}
I20260806 09:11:20.827961 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.789512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:42711 (local address 127.3.69.62:46261)
0806 09:11:20.804371 (+ 14859us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.804380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.807827 (+ 3447us) server_negotiation.cc:407] Connection header received
0806 09:11:20.808776 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.808782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.808886 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.808997 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:20.810681 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.811820 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.814214 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.814846 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.814966 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.820106 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:20.820115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:20.820120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:20.820162 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:20.823966 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.824983 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.824991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.824996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.825166 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.825673 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.825681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.825688 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.826061 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:20.826230 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.827142 (+ 912us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.827443 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14712,"thread_start_us":14873,"threads_started":1}
I20260806 09:11:20.828699 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.789376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:37583 (local address 127.3.69.60:35943)
0806 09:11:20.806387 (+ 17011us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.806393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.806412 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:11:20.806491 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.806496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.806582 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.806694 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:20.809271 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.810494 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.813172 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.813902 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.814047 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.818994 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:20.819001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:20.819008 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:20.819057 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:20.823651 (+ 4594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.824759 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.824764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.824768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.824945 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.825518 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.825525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.825531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.825905 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:20.826066 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.828087 (+ 2021us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.828309 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16881,"thread_start_us":16837,"threads_started":1}
I20260806 09:11:20.829339 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.790392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.3.69.3:36007)
0806 09:11:20.798576 (+ 8184us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:20.798587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:20.802824 (+ 4237us) server_negotiation.cc:407] Connection header received
0806 09:11:20.803034 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:20.803046 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:20.803390 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:20.803621 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:20.806396 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.808044 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.810039 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.810925 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.811026 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.820488 (+ 9462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:20.820506 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:20.820526 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:20.820599 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:20.824881 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.825801 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.825810 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.825812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.825964 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:20.826509 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:20.826514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:20.826516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:20.826947 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:20.827309 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:20.827913 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 09:11:20.828698 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7950,"thread_start_us":7744,"threads_started":1}
I20260806 09:11:20.829645 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.793517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.3.69.2:36317)
0806 09:11:20.808603 (+ 15086us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:20.808620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:20.808808 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:20.809625 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:20.809631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:20.810208 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:20.810579 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.810590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.812064 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.812068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.814044 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.814059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.815254 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.815261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.815466 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.816115 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:20.816142 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:20.819677 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:20.824946 (+ 5269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.824951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.824954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.825471 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.825921 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.825925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.825931 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.826110 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.826843 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:20.826847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:20.826977 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 09:11:20.828115 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0806 09:11:20.828368 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14858,"mutex_wait_us":92,"thread_start_us":14217,"threads_started":1}
I20260806 09:11:20.829730 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.788540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.3.69.2:37583)
0806 09:11:20.800760 (+ 12220us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:20.800835 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:20.805604 (+ 4769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:20.807236 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:20.807252 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:20.808083 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:20.809058 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.809091 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.810680 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.810685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.812947 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.812966 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.814378 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.814387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.814711 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.815475 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:20.815511 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:20.818739 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:20.823846 (+ 5107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.823855 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.823873 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.824605 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.825093 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.825097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.825100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.825364 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.826138 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:20.826149 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:20.826624 (+ 475us) client_negotiation.cc:769] Sending connection context
0806 09:11:20.826937 (+ 313us) client_negotiation.cc:240] Negotiation successful
0806 09:11:20.827534 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11758,"mutex_wait_us":6,"thread_start_us":19581,"threads_started":1}
I20260806 09:11:20.829691 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:20.788585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.2:42711)
0806 09:11:20.807674 (+ 19089us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:20.807706 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:20.808594 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:20.809335 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:20.809339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:20.810026 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:20.810454 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.810495 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.812034 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.812040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.814045 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:20.814059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:20.815669 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:20.815678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:20.815868 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:20.816500 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:20.816521 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:20.819905 (+ 3384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:20.824155 (+ 4250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.824162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.824168 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.824827 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.825328 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:20.825332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:20.825336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:20.825489 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:20.826335 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:20.826339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:20.826647 (+ 308us) client_negotiation.cc:769] Sending connection context
0806 09:11:20.828310 (+ 1663us) client_negotiation.cc:240] Negotiation successful
0806 09:11:20.828715 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18869,"mutex_wait_us":873,"spinlock_wait_cycles":40192,"thread_start_us":10836,"threads_started":1}
I20260806 09:11:20.831571 13632 heartbeater.cc:347] Connected to a master server at 127.3.69.60:35943
I20260806 09:11:20.831619 13635 heartbeater.cc:347] Connected to a master server at 127.3.69.61:46759
I20260806 09:11:20.831835 13634 heartbeater.cc:347] Connected to a master server at 127.3.69.62:46261
I20260806 09:11:20.831966 13634 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:20.831971 13635 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:20.831974 13632 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:20.832921 13635 heartbeater.cc:511] Master 127.3.69.61:46759 requested a full tablet report, sending...
I20260806 09:11:20.833743 12686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:20.834091 13632 heartbeater.cc:511] Master 127.3.69.60:35943 requested a full tablet report, sending...
I20260806 09:11:20.834707 13634 heartbeater.cc:511] Master 127.3.69.62:46261 requested a full tablet report, sending...
I20260806 09:11:20.834995 12911 ts_manager.cc:194] Registered new tserver with Master: e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2:40105)
I20260806 09:11:20.835785 12076 ts_manager.cc:194] Re-registered known tserver with Master: e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2:40105)
I20260806 09:11:20.836184 13158 ts_manager.cc:194] Registered new tserver with Master: e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2:40105)
I20260806 09:11:20.837142 12076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:42711
I20260806 09:11:20.917354 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 12527
I20260806 09:11:20.943244 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:36007
--local_ip_for_outbound_sockets=127.3.69.3
--tserver_master_addrs=127.3.69.62:46261
--webserver_port=38769
--webserver_interface=127.3.69.3
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab
--principal=oryx/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759,127.3.69.60:35943 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:11:21.203222 13419 heartbeater.cc:503] Master 127.3.69.62:46261 was elected leader, sending a full tablet report...
W20260806 09:11:21.284548 13652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:21.284914 13652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:21.285058 13652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:21.296622 13652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:21.296757 13652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:21.296942 13652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:21.306941 13652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab
--principal=oryx/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:36007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=38769
--tserver_master_addrs=127.3.69.62:46261,127.3.69.61:46759,127.3.69.60:35943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:21.308282 13652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:21.309648 13652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:21.323558 13658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:21.323599 13660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:21.323623 13663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:21.324542 13652 server_base.cc:1034] running on GCE node
I20260806 09:11:21.325096 13652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:21.325945 13652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:21.327242 13652 hybrid_clock.cc:648] HybridClock initialized: now 1786007481327154 us; error 55 us; skew 500 ppm
I20260806 09:11:21.327643 13652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:21 dist-test-slave-w4v5 krb5kdc[12015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007481, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:21.335922 13652 init.cc:377] Logged in from keytab as oryx/127.3.69.3@KRBTEST.COM (short username oryx)
I20260806 09:11:21.337387 13652 webserver.cc:460] Webserver started at http://127.3.69.3:38769/ using document root <none> and password file <none>
I20260806 09:11:21.338115 13652 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:21.338277 13652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:21.344416 13652 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:11:21.347438 13673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:21.349018 13652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:11:21.349225 13652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "4e97f231174d4f2f92a35db0df1fad6c"
format_stamp: "Formatted at 2026-08-06 09:11:18 on dist-test-slave-w4v5"
server_key: "40d65929439c0e3b4e192a2454e3aa39"
server_key_iv: "665342c967cac4264764fa65ab50a27a"
server_key_version: "encryptionkey@0"
I20260806 09:11:21.350291 13652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:21.380062 13652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:21.384498 13652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:21.384806 13652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:21.385829 13652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:21.387378 13652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:21.387497 13652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:21.387632 13652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:21.387740 13652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:21.422813 13652 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:36007
I20260806 09:11:21.422902 13834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:36007 every 8 connection(s)
I20260806 09:11:21.424350 13652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:21.429162 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 13652
Aug 06 09:11:21 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007481, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.60@KRBTEST.COM
Aug 06 09:11:21 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007481, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.61@KRBTEST.COM
Aug 06 09:11:21 dist-test-slave-w4v5 krb5kdc[12015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007481, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:21.474216 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.427329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:43457 (local address 127.3.69.60:35943)
0806 09:11:21.443184 (+ 15855us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.443325 (+ 141us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.444525 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 09:11:21.448425 (+ 3900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.448434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.448552 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.448655 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:21.451812 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.453312 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.457008 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.457907 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.458018 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.463636 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:21.463643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:21.463647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:21.463689 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:21.467656 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.469488 (+ 1832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.469494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.469498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.469648 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.470266 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.470273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.470276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.470680 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:21.470823 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.473594 (+ 2771us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.473834 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15660,"thread_start_us":15884,"threads_started":1}
I20260806 09:11:21.476089 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.428073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.3.69.3:43457)
0806 09:11:21.444267 (+ 16194us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:21.444351 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:21.446928 (+ 2577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:21.449282 (+ 2354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:21.449299 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:21.450358 (+ 1059us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:21.451533 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.451565 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.453624 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.453632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.456771 (+ 3139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.456794 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.458391 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.458399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.458694 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.460036 (+ 1342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:21.460087 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:21.463364 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:21.468582 (+ 5218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:21.468591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:21.468610 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:21.469336 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:21.469815 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:21.469819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:21.469823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:21.470117 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:21.470938 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:21.470947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:21.471490 (+ 543us) client_negotiation.cc:769] Sending connection context
0806 09:11:21.474720 (+ 3230us) client_negotiation.cc:240] Negotiation successful
0806 09:11:21.474986 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15873,"thread_start_us":15602,"threads_started":1}
I20260806 09:11:21.476929 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.429186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:49847 (local address 127.3.69.61:46759)
0806 09:11:21.453829 (+ 24643us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.453838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.453862 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:11:21.453935 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.453943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.454041 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.457443 (+ 3402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:21.457639 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.458860 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.460917 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.461619 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.463268 (+ 1649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.466712 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:21.466719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:21.466724 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:21.466772 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:21.471480 (+ 4708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.472592 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.472600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.472604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.472778 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.473271 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.473275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.473278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.473627 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:21.476092 (+ 2465us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.476254 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.476477 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24455,"thread_start_us":24974,"threads_started":1}
I20260806 09:11:21.477283 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.428663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.3.69.3:49847)
0806 09:11:21.447690 (+ 19027us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:21.447884 (+ 194us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:21.448080 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:21.455997 (+ 7917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:21.456005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:21.456840 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:21.457185 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.457208 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.459048 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.459057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.460710 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.460728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.462719 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.462727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.462908 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.463602 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:21.463628 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:21.466468 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:21.471643 (+ 5175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:21.471648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:21.471652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:21.472417 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:21.472936 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:21.472941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:21.472945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:21.473108 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:21.473770 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:21.473776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:21.473909 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 09:11:21.474055 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 09:11:21.474609 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18868,"spinlock_wait_cycles":21248,"thread_start_us":31647,"threads_started":1}
I20260806 09:11:21.480527 13836 heartbeater.cc:347] Connected to a master server at 127.3.69.60:35943
I20260806 09:11:21.480566 13838 heartbeater.cc:347] Connected to a master server at 127.3.69.61:46759
I20260806 09:11:21.480870 13836 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:21.480875 13838 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:21.481582 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.427346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:34597 (local address 127.3.69.62:46261)
0806 09:11:21.442723 (+ 15377us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.442734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.460175 (+ 17441us) server_negotiation.cc:407] Connection header received
0806 09:11:21.460373 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.460378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.460465 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.460573 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:21.461935 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.463034 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.465899 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.466546 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.466662 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.473559 (+ 6897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:21.473567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:21.473574 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:21.473623 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:21.477546 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.478904 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.478910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.478913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.479065 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.479876 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.479881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.479884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.480265 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:21.480412 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.480985 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.481219 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15173,"thread_start_us":15394,"threads_started":1}
I20260806 09:11:21.481931 13838 heartbeater.cc:511] Master 127.3.69.61:46759 requested a full tablet report, sending...
I20260806 09:11:21.482537 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.428073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.3:34597)
0806 09:11:21.460058 (+ 31985us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:21.460082 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:21.460233 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:21.460658 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:21.460662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:21.461361 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:21.461781 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.461796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.463555 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.463568 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.465748 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.465768 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.467368 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.467378 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.467634 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.468304 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:21.468328 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:21.473358 (+ 5030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:21.478250 (+ 4892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:21.478255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:21.478261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:21.478789 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:21.479513 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:21.479520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:21.479524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:21.479689 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:21.480489 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:21.480495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:21.480674 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 09:11:21.481823 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0806 09:11:21.482106 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31783,"mutex_wait_us":3137,"thread_start_us":18544,"threads_started":1}
I20260806 09:11:21.483052 13836 heartbeater.cc:511] Master 127.3.69.60:35943 requested a full tablet report, sending...
I20260806 09:11:21.483867 12911 ts_manager.cc:194] Registered new tserver with Master: 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007)
I20260806 09:11:21.484349 13158 ts_manager.cc:194] Registered new tserver with Master: 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007)
I20260806 09:11:21.485044 13837 heartbeater.cc:347] Connected to a master server at 127.3.69.62:46261
I20260806 09:11:21.485177 13837 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:21.485608 13837 heartbeater.cc:511] Master 127.3.69.62:46261 requested a full tablet report, sending...
I20260806 09:11:21.487111 12076 ts_manager.cc:194] Re-registered known tserver with Master: 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007)
I20260806 09:11:21.488340 12076 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:34597
I20260806 09:11:21.496749 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.446992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.3.69.62:46261)
0806 09:11:21.473899 (+ 26907us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.473905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.473922 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:11:21.473975 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.473979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.474055 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.474164 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:21.476924 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.478057 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.484270 (+ 6213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.484943 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.485072 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.489309 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:21.489315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:21.489319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:21.489355 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:21.493160 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.493918 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.493923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.493927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.494046 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.494429 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.494434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.494438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.494800 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:21.495095 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.496111 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.496293 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26756,"thread_start_us":29757,"threads_started":1}
I20260806 09:11:21.497438 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.453314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.3.69.60:35943)
0806 09:11:21.462887 (+ 9573us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.462893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.466464 (+ 3571us) server_negotiation.cc:407] Connection header received
0806 09:11:21.466532 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.466536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.466642 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.466917 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:21.477252 (+ 10335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.478154 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.482712 (+ 4558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.483502 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.483629 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.490542 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:21.490549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:21.490554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:21.490591 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:21.493684 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.494521 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.494527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.494530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.494699 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.495171 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.495176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.495179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.495528 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:21.496621 (+ 1093us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.496792 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.497022 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9443,"thread_start_us":9861,"threads_started":1}
I20260806 09:11:21.497898 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.454398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.3.69.61:46759)
0806 09:11:21.459455 (+ 5057us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.459461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.459478 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:11:21.459525 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.459530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.459608 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.459711 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:21.478693 (+ 18982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.482543 (+ 3850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.486707 (+ 4164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.487359 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.487451 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.491119 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:21.491133 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:21.491140 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:21.491183 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:21.494521 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.495384 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.495389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.495393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.495526 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.496170 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.496175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.496178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.496516 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:21.496813 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.497396 (+ 583us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.497631 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4910,"thread_start_us":4986,"threads_started":1}
I20260806 09:11:21.501060 12076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:11:21.508394 12076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:21.548161 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.529699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.3.69.2:40105)
0806 09:11:21.540940 (+ 11241us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.540954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.540995 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:11:21.541131 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.541145 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.541518 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.541784 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:21.542519 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.543808 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.546058 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.546928 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.547113 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.547339 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.547605 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.547790 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10990,"thread_start_us":11314,"threads_started":1}
I20260806 09:11:21.548384 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.529118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:40105 (local address 127.0.0.1:58756)
0806 09:11:21.529722 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:21.529745 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:21.529865 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:21.541794 (+ 11929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:21.541802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:21.541842 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:21.542289 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.542300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.543987 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.543991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.545894 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.545906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.547186 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.547193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.547404 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.547430 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 09:11:21.547557 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 09:11:21.547837 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":99,"thread_start_us":795,"threads_started":1}
I20260806 09:11:21.552183 13533 tablet_service.cc:1467] Processing CreateTablet for tablet 49aa097c02b64c93b38aa74025a38638 (DEFAULT_TABLE table=test-table [id=48a5b93899d5464aa1aec9e6154dfc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:21.553396 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49aa097c02b64c93b38aa74025a38638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:21.554205 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.529757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:36007 (local address 127.0.0.1:43386)
0806 09:11:21.539878 (+ 10121us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:21.539901 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:21.540287 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:21.543770 (+ 3483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:21.543778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:21.543814 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:21.544389 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.544406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.546998 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.547006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.550086 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.550108 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.553005 (+ 2897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.553014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.553286 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.553342 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 09:11:21.553455 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 09:11:21.553641 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9995,"mutex_wait_us":78,"thread_start_us":9679,"threads_started":1}
I20260806 09:11:21.556334 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.530429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.3.69.3:36007)
0806 09:11:21.542687 (+ 12258us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.542696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.542736 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:11:21.542859 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.542871 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.543182 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.543422 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:21.544712 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.546797 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.550346 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.552419 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.552582 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.552834 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.555883 (+ 3049us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.556073 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12035,"thread_start_us":11785,"threads_started":1}
I20260806 09:11:21.559808 13741 tablet_service.cc:1467] Processing CreateTablet for tablet 49aa097c02b64c93b38aa74025a38638 (DEFAULT_TABLE table=test-table [id=48a5b93899d5464aa1aec9e6154dfc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:21.561100 13741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49aa097c02b64c93b38aa74025a38638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:21.562446 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.527708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56838)
0806 09:11:21.528311 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:21.528339 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:21.528473 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:21.555514 (+ 27041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:21.555518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:21.555545 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:21.555893 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.555906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.557778 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.557783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.559855 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.559872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.561702 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.561708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.561875 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.561909 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:11:21.561978 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 09:11:21.562114 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":357,"threads_started":1}
I20260806 09:11:21.563475 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.527972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.3.69.1:45171)
0806 09:11:21.554927 (+ 26955us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.554935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.554964 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:11:21.555054 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.555059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.555315 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.555523 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:21.556115 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.557606 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.560040 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.561462 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.562698 (+ 1236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.562893 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.562972 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.563162 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26803,"thread_start_us":26708,"threads_started":1}
I20260806 09:11:21.568252 13322 tablet_service.cc:1467] Processing CreateTablet for tablet 49aa097c02b64c93b38aa74025a38638 (DEFAULT_TABLE table=test-table [id=48a5b93899d5464aa1aec9e6154dfc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:21.568982 13882 tablet_bootstrap.cc:492] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Bootstrap starting.
I20260806 09:11:21.569726 13322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49aa097c02b64c93b38aa74025a38638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:21.572054 13882 tablet_bootstrap.cc:654] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:21.573074 13882 log.cc:824] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:21.577427 13886 tablet_bootstrap.cc:492] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Bootstrap starting.
I20260806 09:11:21.582322 13886 tablet_bootstrap.cc:654] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:21.583655 13887 tablet_bootstrap.cc:492] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Bootstrap starting.
I20260806 09:11:21.583809 13886 log.cc:824] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:21.585496 13882 tablet_bootstrap.cc:492] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: No bootstrap required, opened a new log
I20260806 09:11:21.585809 13882 ts_tablet_manager.cc:1397] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260806 09:11:21.587248 13886 tablet_bootstrap.cc:492] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: No bootstrap required, opened a new log
I20260806 09:11:21.587352 13887 tablet_bootstrap.cc:654] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:21.587514 13886 ts_tablet_manager.cc:1397] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 09:11:21.588663 13887 log.cc:824] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:21.591620 13887 tablet_bootstrap.cc:492] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: No bootstrap required, opened a new log
I20260806 09:11:21.591876 13887 ts_tablet_manager.cc:1397] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260806 09:11:21.593957 13882 raft_consensus.cc:348] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.594326 13882 raft_consensus.cc:374] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:21.594458 13882 raft_consensus.cc:729] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d35582e97a4bfea3165b8f33a3606b, State: Initialized, Role: FOLLOWER
I20260806 09:11:21.595124 13882 consensus_queue.cc:260] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.596865 13882 ts_tablet_manager.cc:1428] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260806 09:11:21.597132 13634 heartbeater.cc:503] Master 127.3.69.62:46261 was elected leader, sending a full tablet report...
I20260806 09:11:21.598053 13886 raft_consensus.cc:348] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.598501 13886 raft_consensus.cc:374] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:21.598595 13886 raft_consensus.cc:729] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e97f231174d4f2f92a35db0df1fad6c, State: Initialized, Role: FOLLOWER
I20260806 09:11:21.599483 13886 consensus_queue.cc:260] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.601750 13886 ts_tablet_manager.cc:1428] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 09:11:21.602267 13837 heartbeater.cc:503] Master 127.3.69.62:46261 was elected leader, sending a full tablet report...
I20260806 09:11:21.603367 13887 raft_consensus.cc:348] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.603838 13887 raft_consensus.cc:374] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:21.603971 13887 raft_consensus.cc:729] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59aeb9db55843f8af2e0a3197e5ea96, State: Initialized, Role: FOLLOWER
I20260806 09:11:21.604621 13887 consensus_queue.cc:260] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.609951 13887 ts_tablet_manager.cc:1428] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
W20260806 09:11:21.657719 13422 tablet.cc:2369] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:11:21.677425 13839 tablet.cc:2369] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:21.702346 13893 raft_consensus.cc:482] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:21.702656 13893 raft_consensus.cc:504] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.704492 13893 leader_election.cc:288] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2:40105), 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007)
I20260806 09:11:21.711652 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.704814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:39805 (local address 127.3.69.2:40105)
0806 09:11:21.705051 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.705058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.705796 (+ 738us) server_negotiation.cc:407] Connection header received
0806 09:11:21.706002 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.706008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.706088 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.706158 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:21.706812 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.707617 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.709801 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.710608 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.710765 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.710880 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.711244 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.711368 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:11:21.711651 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.704936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:42141 (local address 127.3.69.3:36007)
0806 09:11:21.705152 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.705160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.705866 (+ 706us) server_negotiation.cc:407] Connection header received
0806 09:11:21.706052 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.706055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.706113 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.706225 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:21.706880 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.707639 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.709850 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.710616 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.710787 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.710899 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.711277 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.711396 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 09:11:21.711784 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.704709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:40105 (local address 127.3.69.1:39805)
0806 09:11:21.705646 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:21.705662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:21.705834 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:21.706223 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:21.706228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:21.706252 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:21.706656 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.706668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.707762 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.707767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.709648 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.709660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.710855 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.710861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.711016 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.711065 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 09:11:21.711192 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 09:11:21.711338 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":97,"thread_start_us":583,"threads_started":1}
I20260806 09:11:21.712025 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.704736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:36007 (local address 127.3.69.1:42141)
0806 09:11:21.705686 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:21.705702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:21.705915 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:21.706259 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:21.706264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:21.706289 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:21.706677 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.706688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.707808 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.707812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.709670 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:21.709682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.710893 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.710902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:21.711047 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.711113 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 09:11:21.711232 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 09:11:21.711359 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":19,"thread_start_us":591,"threads_started":1}
I20260806 09:11:21.712889 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49aa097c02b64c93b38aa74025a38638" candidate_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d35582e97a4bfea3165b8f33a3606b" is_pre_election: true
I20260806 09:11:21.713099 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49aa097c02b64c93b38aa74025a38638" candidate_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e97f231174d4f2f92a35db0df1fad6c" is_pre_election: true
I20260806 09:11:21.713408 13562 raft_consensus.cc:2454] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59aeb9db55843f8af2e0a3197e5ea96 in term 0.
I20260806 09:11:21.713639 13775 raft_consensus.cc:2454] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59aeb9db55843f8af2e0a3197e5ea96 in term 0.
I20260806 09:11:21.714097 13263 leader_election.cc:302] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c59aeb9db55843f8af2e0a3197e5ea96, e4d35582e97a4bfea3165b8f33a3606b; no voters:
I20260806 09:11:21.714471 13893 raft_consensus.cc:2781] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:21.714622 13893 raft_consensus.cc:482] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:21.714730 13893 raft_consensus.cc:3037] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:21.716948 13893 raft_consensus.cc:504] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.717608 13893 leader_election.cc:288] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [CANDIDATE]: Term 1 election: Requested vote from peers e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2:40105), 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007)
I20260806 09:11:21.717933 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49aa097c02b64c93b38aa74025a38638" candidate_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d35582e97a4bfea3165b8f33a3606b"
I20260806 09:11:21.718150 13562 raft_consensus.cc:3037] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:21.718127 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49aa097c02b64c93b38aa74025a38638" candidate_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e97f231174d4f2f92a35db0df1fad6c"
I20260806 09:11:21.718336 13775 raft_consensus.cc:3037] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:21.720742 13562 raft_consensus.cc:2454] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59aeb9db55843f8af2e0a3197e5ea96 in term 1.
I20260806 09:11:21.720762 13775 raft_consensus.cc:2454] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59aeb9db55843f8af2e0a3197e5ea96 in term 1.
I20260806 09:11:21.721240 13261 leader_election.cc:302] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e97f231174d4f2f92a35db0df1fad6c, c59aeb9db55843f8af2e0a3197e5ea96; no voters:
I20260806 09:11:21.721568 13893 raft_consensus.cc:2781] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:21.722079 13893 raft_consensus.cc:686] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 LEADER]: Becoming Leader. State: Replica: c59aeb9db55843f8af2e0a3197e5ea96, State: Running, Role: LEADER
I20260806 09:11:21.722550 13893 consensus_queue.cc:237] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } }
I20260806 09:11:21.728037 12076 catalog_manager.cc:5495] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 reported cstate change: term changed from 0 to 1, leader changed from <none> to c59aeb9db55843f8af2e0a3197e5ea96 (127.3.69.1). New cstate: current_term: 1 leader_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
W20260806 09:11:21.790148 13636 tablet.cc:2369] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:21.793358 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.787551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.3.69.1:45171)
0806 09:11:21.787752 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.787757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.787774 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:11:21.787863 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.787867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.787956 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.788098 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:21.788463 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.789196 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.791150 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.791575 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.791685 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.792234 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:21.792718 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:21.792832 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.792983 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.793054 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:11:21.802297 13562 raft_consensus.cc:1261] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 1 FOLLOWER]: Refusing update from remote peer c59aeb9db55843f8af2e0a3197e5ea96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:21.802302 13775 raft_consensus.cc:1261] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 1 FOLLOWER]: Refusing update from remote peer c59aeb9db55843f8af2e0a3197e5ea96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:21.803357 13896 consensus_queue.cc:1035] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:21.803727 13893 consensus_queue.cc:1035] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:21.818464 13900 mvcc.cc:205] Tried to move back new op lower bound from 7315486645452648448 to 7315486645140201472. Current Snapshot: MvccSnapshot[applied={T|T < 7315486645452648448}]
I20260806 09:11:21.844110 12076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:21.844405 12076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:21.851881 12076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f4dfd6248ab451497efc55c64e34050: Sending DeleteTablet for 3 replicas of tablet 49aa097c02b64c93b38aa74025a38638
I20260806 09:11:21.853284 13533 tablet_service.cc:1514] Processing DeleteTablet for tablet 49aa097c02b64c93b38aa74025a38638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:21 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:58756
I20260806 09:11:21.853487 13322 tablet_service.cc:1514] Processing DeleteTablet for tablet 49aa097c02b64c93b38aa74025a38638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:21 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:56838
I20260806 09:11:21.854223 13741 tablet_service.cc:1514] Processing DeleteTablet for tablet 49aa097c02b64c93b38aa74025a38638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:21 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:43386
I20260806 09:11:21.864377 13921 tablet_replica.cc:330] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: stopping tablet replica
I20260806 09:11:21.865200 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 12034
I20260806 09:11:21.866348 12123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:21.866401 13921 raft_consensus.cc:2229] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:21.870507 13921 raft_consensus.cc:2258] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:21.871629 13920 tablet_replica.cc:330] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: stopping tablet replica
I20260806 09:11:21.872241 13920 raft_consensus.cc:2229] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:11:21.872891 13921 ts_tablet_manager.cc:1905] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:21.875300 13920 raft_consensus.cc:2258] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:21.879639 13920 ts_tablet_manager.cc:1905] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:21.884119 13915 tablet_replica.cc:330] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: stopping tablet replica
I20260806 09:11:21.884759 13915 raft_consensus.cc:2229] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:21.885469 13915 raft_consensus.cc:2258] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:21.888211 13915 ts_tablet_manager.cc:1905] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:21.890127 13921 ts_tablet_manager.cc:1918] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:21.890337 13921 log.cc:1196] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/49aa097c02b64c93b38aa74025a38638
I20260806 09:11:21.890930 13921 ts_tablet_manager.cc:1939] T 49aa097c02b64c93b38aa74025a38638 P 4e97f231174d4f2f92a35db0df1fad6c: Deleting consensus metadata
I20260806 09:11:21.893289 13920 ts_tablet_manager.cc:1918] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:21.893486 13920 log.cc:1196] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/49aa097c02b64c93b38aa74025a38638
I20260806 09:11:21.894305 13920 ts_tablet_manager.cc:1939] T 49aa097c02b64c93b38aa74025a38638 P c59aeb9db55843f8af2e0a3197e5ea96: Deleting consensus metadata
I20260806 09:11:21.901191 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.857911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.3.69.61:46759)
0806 09:11:21.858174 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.858183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.863870 (+ 5687us) server_negotiation.cc:407] Connection header received
0806 09:11:21.866575 (+ 2705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.866584 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.866706 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.866796 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:21.873142 (+ 6346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.874468 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.877738 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.878602 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.878734 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.888559 (+ 9825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:21.888569 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:21.888577 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:21.888638 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:21.894490 (+ 5852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.895442 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.895448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.895452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.895600 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.899251 (+ 3651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.899257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.899260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.899644 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:21.899917 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.900574 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.900843 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:11:21.904881 13915 ts_tablet_manager.cc:1918] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:21.905171 13915 log.cc:1196] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/49aa097c02b64c93b38aa74025a38638
I20260806 09:11:21.905851 13915 ts_tablet_manager.cc:1939] T 49aa097c02b64c93b38aa74025a38638 P e4d35582e97a4bfea3165b8f33a3606b: Deleting consensus metadata
I20260806 09:11:21.919327 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:21.863835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.3.69.60:35943)
0806 09:11:21.864063 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:21.864069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:21.902354 (+ 38285us) server_negotiation.cc:407] Connection header received
0806 09:11:21.902625 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:21.902631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:21.902739 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:21.902823 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:21.904824 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.905969 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.907888 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:21.908564 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:21.908708 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:21.912311 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:21.912320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:21.912327 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:21.912378 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:21.916005 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.916809 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.916815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.916818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.916968 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:21.917406 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:21.917411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:21.917415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:21.917776 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:21.918028 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:21.918575 (+ 547us) server_negotiation.cc:299] Negotiation successful
0806 09:11:21.918889 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:11:22.031742 12056 catalog_manager.cc:4841] TS e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2:40105): tablet 49aa097c02b64c93b38aa74025a38638 (table test-table [id=48a5b93899d5464aa1aec9e6154dfc1a]) successfully deleted
I20260806 09:11:22.033542 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 12034
W20260806 09:11:22.072012 13261 connection.cc:535] server connection from 127.0.0.1:56838 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 09:11:22.098277 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:22.078261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.3.69.61:46759)
0806 09:11:22.078701 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:22.078710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:22.080013 (+ 1303us) server_negotiation.cc:407] Connection header received
0806 09:11:22.080870 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:22.080877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:22.080986 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:22.081098 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:22.083352 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:22.084542 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:22.086766 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:22.088493 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:22.088606 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:22.092016 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:22.092021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:22.092025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:22.092065 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:22.094956 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:22.096048 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:22.096054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:22.096057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:22.096219 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:22.096707 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:22.096713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:22.096716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:22.097123 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:22.097371 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:22.097683 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 09:11:22.097903 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260806 09:11:22.099516 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:22.078028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.3.69.60:35943)
0806 09:11:22.078514 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:22.078524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:22.080638 (+ 2114us) server_negotiation.cc:407] Connection header received
0806 09:11:22.081498 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:22.081507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:22.081601 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:22.081728 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:22.083809 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:22.084528 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:22.087178 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:22.087831 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:22.087957 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:22.091679 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:22.091686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:22.091691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:22.091741 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:22.095238 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:22.096105 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:22.096110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:22.096114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:22.096246 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:22.097451 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:22.097456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:22.097460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:22.097963 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:22.098206 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:22.098903 (+ 697us) server_negotiation.cc:299] Negotiation successful
0806 09:11:22.099162 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260806 09:11:22.824484 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:22.823611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.2:58719)
0806 09:11:22.824127 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:22.824295 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":263,"threads_started":1}
W20260806 09:11:22.825030 13634 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:46261 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
I20260806 09:11:22.829035 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:22.828180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.3:39673)
0806 09:11:22.828708 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:22.828854 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":272,"threads_started":1}
W20260806 09:11:22.829439 13837 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:46261 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
I20260806 09:11:22.853737 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:22.852993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.1:54687)
0806 09:11:22.853387 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:22.853568 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":195,"threads_started":1}
W20260806 09:11:22.854146 13419 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:46261 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
I20260806 09:11:23.351794 13937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f4dfd6248ab451497efc55c64e34050)
I20260806 09:11:23.352130 13937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } }
I20260806 09:11:23.353688 13937 leader_election.cc:288] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f4dfd6248ab451497efc55c64e34050 (127.3.69.62:46261), a72f9d9f4c654884a17110a56b53d460 (127.3.69.61:46759)
I20260806 09:11:23.354661 13938 raft_consensus.cc:482] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7f4dfd6248ab451497efc55c64e34050)
I20260806 09:11:23.355049 13938 raft_consensus.cc:504] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } }
I20260806 09:11:23.355671 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.353807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44508)
0806 09:11:23.354937 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.355193 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":674,"threads_started":1}
W20260806 09:11:23.355932 13140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
I20260806 09:11:23.357820 13938 leader_election.cc:288] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f4dfd6248ab451497efc55c64e34050 (127.3.69.62:46261), fc23bdd3625d48268b028d395e9659b7 (127.3.69.60:35943)
I20260806 09:11:23.358457 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.356741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44524)
0806 09:11:23.357934 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.358135 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":620,"threads_started":1}
W20260806 09:11:23.358691 12891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
I20260806 09:11:23.364246 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.357779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44528)
0806 09:11:23.363961 (+ 6182us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.364059 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6074}
I20260806 09:11:23.365000 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.354496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.3.69.61:46759)
0806 09:11:23.355253 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.355260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.355789 (+ 529us) server_negotiation.cc:407] Connection header received
0806 09:11:23.356090 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.356098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.356182 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.356312 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:23.356954 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.358183 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.361336 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.363232 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.363391 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.363535 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.364326 (+ 791us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.364506 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":334,"threads_started":1}
I20260806 09:11:23.365458 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.354039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:46759 (local address 127.0.0.1:59056)
0806 09:11:23.355678 (+ 1639us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.355698 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:23.355904 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:23.356337 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:23.356344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:23.356375 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:23.356747 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.356761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.358374 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.358383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.361069 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.361091 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.363688 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.363697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.363930 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.363964 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:11:23.364070 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 09:11:23.364199 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"thread_start_us":1352,"threads_started":1}
I20260806 09:11:23.366297 12926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc23bdd3625d48268b028d395e9659b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a72f9d9f4c654884a17110a56b53d460" is_pre_election: true
I20260806 09:11:23.366719 12926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc23bdd3625d48268b028d395e9659b7 in term 1.
I20260806 09:11:23.367465 13137 leader_election.cc:302] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9d9f4c654884a17110a56b53d460, fc23bdd3625d48268b028d395e9659b7; no voters:
I20260806 09:11:23.367928 13937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 09:11:23.368083 13937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7f4dfd6248ab451497efc55c64e34050)
I20260806 09:11:23.368221 13937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 1 FOLLOWER]: Advancing to term 2
W20260806 09:11:23.368523 13140 leader_election.cc:334] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f4dfd6248ab451497efc55c64e34050 (127.3.69.62:46261): Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
I20260806 09:11:23.369843 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.368230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44534)
0806 09:11:23.369463 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.369636 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1101}
W20260806 09:11:23.369992 12891 leader_election.cc:334] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7f4dfd6248ab451497efc55c64e34050 (127.3.69.62:46261): Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
I20260806 09:11:23.372205 13937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } }
I20260806 09:11:23.373430 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.373017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44552)
0806 09:11:23.373194 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.373281 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 09:11:23.373853 12926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc23bdd3625d48268b028d395e9659b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a72f9d9f4c654884a17110a56b53d460"
I20260806 09:11:23.374002 13937 leader_election.cc:288] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [CANDIDATE]: Term 2 election: Requested vote from peers 7f4dfd6248ab451497efc55c64e34050 (127.3.69.62:46261), a72f9d9f4c654884a17110a56b53d460 (127.3.69.61:46759)
I20260806 09:11:23.374050 12926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:11:23.375198 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.358007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:35943 (local address 127.0.0.1:60736)
0806 09:11:23.365204 (+ 7197us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.365221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:23.365327 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:23.368443 (+ 3116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:23.368448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:23.368475 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:23.368702 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.368716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.370728 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.370733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.372464 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.372481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.374277 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.374284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.374517 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.374549 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:11:23.374687 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 09:11:23.374839 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7075,"thread_start_us":11166,"threads_started":1}
I20260806 09:11:23.375284 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.374841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44562)
0806 09:11:23.375008 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.375118 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 09:11:23.375531 13140 leader_election.cc:334] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7f4dfd6248ab451497efc55c64e34050 (127.3.69.62:46261): Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
I20260806 09:11:23.376122 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.358395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.3.69.60:35943)
0806 09:11:23.368127 (+ 9732us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.368134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.368158 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:11:23.368217 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.368222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.368290 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.368843 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:23.369167 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.370552 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.372648 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.374065 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.374217 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.374331 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.375185 (+ 854us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.375790 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9591,"thread_start_us":9942,"threads_started":1}
I20260806 09:11:23.376623 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a72f9d9f4c654884a17110a56b53d460" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc23bdd3625d48268b028d395e9659b7" is_pre_election: true
I20260806 09:11:23.376945 13173 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a72f9d9f4c654884a17110a56b53d460 in current term 2: Already voted for candidate fc23bdd3625d48268b028d395e9659b7 in this term.
I20260806 09:11:23.377439 12926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc23bdd3625d48268b028d395e9659b7 in term 2.
I20260806 09:11:23.377544 12890 leader_election.cc:302] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a72f9d9f4c654884a17110a56b53d460; no voters: 7f4dfd6248ab451497efc55c64e34050, fc23bdd3625d48268b028d395e9659b7
I20260806 09:11:23.377864 13137 leader_election.cc:302] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a72f9d9f4c654884a17110a56b53d460, fc23bdd3625d48268b028d395e9659b7; no voters: 7f4dfd6248ab451497efc55c64e34050
I20260806 09:11:23.377972 13938 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:11:23.378273 13937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:11:23.378712 13937 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [term 2 LEADER]: Becoming Leader. State: Replica: fc23bdd3625d48268b028d395e9659b7, State: Running, Role: LEADER
I20260806 09:11:23.379122 13937 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } }
I20260806 09:11:23.380986 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc23bdd3625d48268b028d395e9659b7. Latest consensus state: current_term: 2 leader_uuid: "fc23bdd3625d48268b028d395e9659b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:23.381415 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:23.382157 13951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:23.386153 13951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:23.386982 13951 catalog_manager.cc:1259] Loaded cluster ID: e1c2b4515b97401c906286923abfccca
I20260806 09:11:23.387106 13951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:23.388296 13951 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:23.389161 13951 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Loaded TSK: 0
I20260806 09:11:23.390086 13951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:11:23.431233 13157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:11:23.438352 13157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:23.442919 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.442518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44574)
0806 09:11:23.442675 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.442754 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 09:11:23.442976 12926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [term 2 FOLLOWER]: Refusing update from remote peer fc23bdd3625d48268b028d395e9659b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 09:11:23.444100 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.443682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44584)
0806 09:11:23.443856 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.443929 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 09:11:23.444169 13949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 09:11:23.444563 13140 consensus_peers.cc:487] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 -> Peer 7f4dfd6248ab451497efc55c64e34050 (127.3.69.62:46261): Couldn't send request to peer 7f4dfd6248ab451497efc55c64e34050. Status: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 09:11:23.447249 13938 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc23bdd3625d48268b028d395e9659b7. Latest consensus state: current_term: 2 leader_uuid: "fc23bdd3625d48268b028d395e9659b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:23.447522 13938 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:11:23.448827 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "fc23bdd3625d48268b028d395e9659b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:23.449126 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:23.449338 13938 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "fc23bdd3625d48268b028d395e9659b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7f4dfd6248ab451497efc55c64e34050" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 46261 } } peers { permanent_uuid: "a72f9d9f4c654884a17110a56b53d460" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 46759 } attrs { promote: false } } peers { permanent_uuid: "fc23bdd3625d48268b028d395e9659b7" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 35943 } attrs { promote: false } } }
I20260806 09:11:23.449580 13938 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72f9d9f4c654884a17110a56b53d460 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:11:23.467590 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.457401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56850)
0806 09:11:23.457542 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.457560 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:23.457686 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:23.459273 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:23.459280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:23.459318 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:23.459602 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.459617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.461627 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.461636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.464250 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.464273 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.466450 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.466459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.466776 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.466837 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 09:11:23.466944 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 09:11:23.467173 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260806 09:11:23.469429 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.457531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.3.69.1:45171)
0806 09:11:23.458810 (+ 1279us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.458819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.458845 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:11:23.458938 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.458946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.459048 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.459790 (+ 742us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:23.460012 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.461413 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.464520 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.466225 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.468604 (+ 2379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.468769 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.468835 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.469031 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"thread_start_us":942,"threads_started":1}
I20260806 09:11:23.470579 13322 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 (DEFAULT_TABLE table=test-table [id=ef46df391667410dbeabf608a38bcef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:23.471115 13322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:23.475973 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.458394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:40105 (local address 127.0.0.1:58764)
0806 09:11:23.467832 (+ 9438us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.467854 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:23.468014 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:23.468467 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:23.468475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:23.468509 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:23.468786 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.468800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.470877 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.470886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.473399 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.473414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.475038 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.475046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.475344 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.475396 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 09:11:23.475499 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 09:11:23.475639 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9321,"thread_start_us":11177,"threads_started":1}
I20260806 09:11:23.476001 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.458867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.3.69.2:40105)
0806 09:11:23.462218 (+ 3351us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.462227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.467932 (+ 5705us) server_negotiation.cc:407] Connection header received
0806 09:11:23.468159 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.468168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.468274 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.468961 (+ 687us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:23.469192 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.470665 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.473584 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.474865 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.474999 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.475150 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.475540 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.475685 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3208,"thread_start_us":3101,"threads_started":1}
I20260806 09:11:23.477121 13533 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 (DEFAULT_TABLE table=test-table [id=ef46df391667410dbeabf608a38bcef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:23.477625 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:23.477756 13968 tablet_bootstrap.cc:492] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96: Bootstrap starting.
I20260806 09:11:23.479481 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.457881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:36007 (local address 127.0.0.1:43396)
0806 09:11:23.458094 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.458115 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:23.458239 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:23.473720 (+ 15481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:23.473724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:23.473750 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:23.474062 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.474071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.475203 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.475209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.477066 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.477078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.478755 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.478764 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.478952 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.478990 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:11:23.479061 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 09:11:23.479228 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 09:11:23.479890 13968 tablet_bootstrap.cc:654] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:23.480409 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.457989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.3.69.3:36007)
0806 09:11:23.473342 (+ 15353us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.473350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.473370 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:11:23.473448 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.473456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.473572 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.473700 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:23.474218 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.475006 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.477307 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.478474 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.479734 (+ 1260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.479864 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.479923 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.480061 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15213,"thread_start_us":15848,"threads_started":1}
I20260806 09:11:23.481274 13741 tablet_service.cc:1467] Processing CreateTablet for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 (DEFAULT_TABLE table=test-table [id=ef46df391667410dbeabf608a38bcef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:23.481755 13741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:23.482853 13968 tablet_bootstrap.cc:492] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96: No bootstrap required, opened a new log
I20260806 09:11:23.483042 13968 ts_tablet_manager.cc:1397] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 09:11:23.483733 13970 tablet_bootstrap.cc:492] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: Bootstrap starting.
I20260806 09:11:23.484189 13968 raft_consensus.cc:348] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.484421 13968 raft_consensus.cc:374] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:23.484508 13968 raft_consensus.cc:729] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59aeb9db55843f8af2e0a3197e5ea96, State: Initialized, Role: FOLLOWER
I20260806 09:11:23.484733 13968 consensus_queue.cc:260] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.485391 13970 tablet_bootstrap.cc:654] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:23.486598 13968 ts_tablet_manager.cc:1428] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 09:11:23.487607 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.486409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.1:35169)
0806 09:11:23.487162 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.487317 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"threads_started":1}
I20260806 09:11:23.490098 13417 heartbeater.cc:503] Master 127.3.69.60:35943 was elected leader, sending a full tablet report...
I20260806 09:11:23.491531 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.491069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.1:60769)
0806 09:11:23.491317 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.491405 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 09:11:23.492867 13976 tablet_bootstrap.cc:492] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: Bootstrap starting.
I20260806 09:11:23.494108 13970 tablet_bootstrap.cc:492] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: No bootstrap required, opened a new log
I20260806 09:11:23.494258 13970 ts_tablet_manager.cc:1397] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260806 09:11:23.494707 13976 tablet_bootstrap.cc:654] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:23.495158 13970 raft_consensus.cc:348] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.495499 13970 raft_consensus.cc:374] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:23.495613 13970 raft_consensus.cc:729] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d35582e97a4bfea3165b8f33a3606b, State: Initialized, Role: FOLLOWER
I20260806 09:11:23.495975 13970 consensus_queue.cc:260] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.497037 13970 ts_tablet_manager.cc:1428] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:23.498307 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.497204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.2:58459)
0806 09:11:23.497840 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.498022 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":357,"threads_started":1}
I20260806 09:11:23.499392 13632 heartbeater.cc:503] Master 127.3.69.60:35943 was elected leader, sending a full tablet report...
I20260806 09:11:23.500151 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.499580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.2:42275)
0806 09:11:23.499924 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.500010 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260806 09:11:23.502466 13976 tablet_bootstrap.cc:492] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: No bootstrap required, opened a new log
I20260806 09:11:23.502585 13976 ts_tablet_manager.cc:1397] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 09:11:23.503243 13976 raft_consensus.cc:348] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.503419 13976 raft_consensus.cc:374] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:23.503491 13976 raft_consensus.cc:729] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e97f231174d4f2f92a35db0df1fad6c, State: Initialized, Role: FOLLOWER
I20260806 09:11:23.503723 13976 consensus_queue.cc:260] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.504383 13976 ts_tablet_manager.cc:1428] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 09:11:23.505578 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.504704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.3:46373)
0806 09:11:23.505279 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.505389 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":298,"threads_started":1}
I20260806 09:11:23.506505 13836 heartbeater.cc:503] Master 127.3.69.60:35943 was elected leader, sending a full tablet report...
I20260806 09:11:23.506682 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.506169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.3:59223)
0806 09:11:23.506454 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.506552 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260806 09:11:23.548830 13978 raft_consensus.cc:482] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:23.549084 13978 raft_consensus.cc:504] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.550848 13978 leader_election.cc:288] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c59aeb9db55843f8af2e0a3197e5ea96 (127.3.69.1:45171), 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007)
I20260806 09:11:23.557150 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.551066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.3.69.2:44641)
0806 09:11:23.551299 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.551320 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:23.551467 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:23.552001 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:23.552006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:23.552060 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:23.552372 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.552383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.553382 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.553388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.554909 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.554922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.556372 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.556378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.556559 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.556593 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:11:23.556709 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 09:11:23.556863 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 09:11:23.557170 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.551291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:44641 (local address 127.3.69.1:45171)
0806 09:11:23.551494 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.551502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.551528 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:11:23.551614 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.551619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.551713 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.551826 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:23.552547 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.553240 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.555146 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.556178 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.556310 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.556421 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.556742 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.556883 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:11:23.558187 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.551691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:35681 (local address 127.3.69.3:36007)
0806 09:11:23.551896 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.551901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.552325 (+ 424us) server_negotiation.cc:407] Connection header received
0806 09:11:23.552532 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.552538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.552608 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.552713 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:23.553201 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.554026 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.555994 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.557128 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.557279 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.557387 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.557914 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.558010 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:11:23.558095 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd4bb8bc4054d8bbe0b4e0ba1d970b0" candidate_uuid: "e4d35582e97a4bfea3165b8f33a3606b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" is_pre_election: true
I20260806 09:11:23.558350 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.551471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:36007 (local address 127.3.69.2:35681)
0806 09:11:23.552210 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.552229 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:23.552384 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:23.552723 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:23.552728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:23.552752 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:23.553045 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.553053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.554154 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.554159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.555797 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:23.555810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.557567 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.557571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:23.557736 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.557769 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:11:23.557879 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 09:11:23.558015 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":340,"threads_started":1}
I20260806 09:11:23.558568 13347 raft_consensus.cc:2454] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4d35582e97a4bfea3165b8f33a3606b in term 0.
I20260806 09:11:23.559055 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd4bb8bc4054d8bbe0b4e0ba1d970b0" candidate_uuid: "e4d35582e97a4bfea3165b8f33a3606b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e97f231174d4f2f92a35db0df1fad6c" is_pre_election: true
I20260806 09:11:23.559247 13468 leader_election.cc:302] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c59aeb9db55843f8af2e0a3197e5ea96, e4d35582e97a4bfea3165b8f33a3606b; no voters:
I20260806 09:11:23.559345 13775 raft_consensus.cc:2454] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4d35582e97a4bfea3165b8f33a3606b in term 0.
I20260806 09:11:23.559758 13978 raft_consensus.cc:2781] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:23.559898 13978 raft_consensus.cc:482] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:23.560031 13978 raft_consensus.cc:3037] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:23.562240 13978 raft_consensus.cc:504] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.562808 13978 leader_election.cc:288] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [CANDIDATE]: Term 1 election: Requested vote from peers c59aeb9db55843f8af2e0a3197e5ea96 (127.3.69.1:45171), 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007)
I20260806 09:11:23.563376 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd4bb8bc4054d8bbe0b4e0ba1d970b0" candidate_uuid: "e4d35582e97a4bfea3165b8f33a3606b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59aeb9db55843f8af2e0a3197e5ea96"
I20260806 09:11:23.563421 13775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cd4bb8bc4054d8bbe0b4e0ba1d970b0" candidate_uuid: "e4d35582e97a4bfea3165b8f33a3606b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e97f231174d4f2f92a35db0df1fad6c"
I20260806 09:11:23.563578 13347 raft_consensus.cc:3037] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:23.563642 13775 raft_consensus.cc:3037] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:23.565694 13775 raft_consensus.cc:2454] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d35582e97a4bfea3165b8f33a3606b in term 1.
I20260806 09:11:23.565701 13347 raft_consensus.cc:2454] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d35582e97a4bfea3165b8f33a3606b in term 1.
I20260806 09:11:23.566102 13468 leader_election.cc:302] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c59aeb9db55843f8af2e0a3197e5ea96, e4d35582e97a4bfea3165b8f33a3606b; no voters:
I20260806 09:11:23.566424 13978 raft_consensus.cc:2781] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:23.566888 13978 raft_consensus.cc:686] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 1 LEADER]: Becoming Leader. State: Replica: e4d35582e97a4bfea3165b8f33a3606b, State: Running, Role: LEADER
I20260806 09:11:23.567677 13978 consensus_queue.cc:237] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } }
I20260806 09:11:23.568145 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.567517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.2:52501)
0806 09:11:23.567822 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.567989 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260806 09:11:23.572741 13157 catalog_manager.cc:5495] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b reported cstate change: term changed from 0 to 1, leader changed from <none> to e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2). New cstate: current_term: 1 leader_uuid: "e4d35582e97a4bfea3165b8f33a3606b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d35582e97a4bfea3165b8f33a3606b" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 40105 } health_report { overall_health: HEALTHY } } }
I20260806 09:11:23.612596 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.607790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.3.69.2:40105)
0806 09:11:23.607980 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.607985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.608003 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:11:23.608098 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.608101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.608181 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.608355 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:23.608706 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.609384 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.610722 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.611150 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.611250 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.611604 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:23.612128 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:23.612246 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.612322 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.612388 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 09:11:23.616818 13775 raft_consensus.cc:1261] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 1 FOLLOWER]: Refusing update from remote peer e4d35582e97a4bfea3165b8f33a3606b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:23.617034 13347 raft_consensus.cc:1261] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 FOLLOWER]: Refusing update from remote peer e4d35582e97a4bfea3165b8f33a3606b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:23.617594 13984 consensus_queue.cc:1035] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e97f231174d4f2f92a35db0df1fad6c" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:23.618011 13978 consensus_queue.cc:1035] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59aeb9db55843f8af2e0a3197e5ea96" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 45171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:23.620739 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.620005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.3:33745)
0806 09:11:23.620339 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.620487 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260806 09:11:23.620805 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.620309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.1:51019)
0806 09:11:23.620451 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.620555 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 09:11:23.629004 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.628447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.2:45373)
0806 09:11:23.628730 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.628817 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260806 09:11:23.630277 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.629823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.1:44893)
0806 09:11:23.630047 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.630132 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 09:11:23.634471 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.633985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.2:34941)
0806 09:11:23.634199 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.634298 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 09:11:23.637235 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.636826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.3.69.3:54513)
0806 09:11:23.636988 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.637073 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 09:11:23.644178 13158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:23.644507 13158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:23.649773 13158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc23bdd3625d48268b028d395e9659b7: Sending DeleteTablet for 3 replicas of tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0
I20260806 09:11:23.650868 13741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:23 UTC) from {username='oryx', principal='oryx/127.3.69.60@KRBTEST.COM'} at 127.0.0.1:43396
I20260806 09:11:23.651149 13322 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:23 UTC) from {username='oryx', principal='oryx/127.3.69.60@KRBTEST.COM'} at 127.0.0.1:56850
I20260806 09:11:23.651144 13533 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:23 UTC) from {username='oryx', principal='oryx/127.3.69.60@KRBTEST.COM'} at 127.0.0.1:58764
I20260806 09:11:23.652892 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 13214
I20260806 09:11:23.665659 13998 tablet_replica.cc:330] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: stopping tablet replica
I20260806 09:11:23.665947 13998 raft_consensus.cc:2229] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:23.666316 13998 raft_consensus.cc:2258] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:23.667713 13999 tablet_replica.cc:330] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96: stopping tablet replica
I20260806 09:11:23.667989 13999 raft_consensus.cc:2229] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:23.668102 13998 ts_tablet_manager.cc:1905] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:23.668570 14000 tablet_replica.cc:330] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: stopping tablet replica
I20260806 09:11:23.668769 14000 raft_consensus.cc:2229] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:11:23.669376 14000 raft_consensus.cc:2258] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:23.670186 13999 raft_consensus.cc:2258] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:23.671676 14000 ts_tablet_manager.cc:1905] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:23.680241 13998 ts_tablet_manager.cc:1918] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:23.681420 13962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.654686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.3.69.1:45171)
0806 09:11:23.654910 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.654918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.664440 (+ 9522us) server_negotiation.cc:407] Connection header received
0806 09:11:23.664693 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.664702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.664813 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.664955 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:23.667029 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.668316 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.670066 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.670817 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.670952 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.674342 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:23.674359 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:23.674380 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:23.674421 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:23.678334 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:23.679171 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:23.679181 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:23.679185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:23.679337 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:23.679744 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:23.679749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:23.679752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:23.680242 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:23.680616 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.680833 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.681048 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:11:23.681488 13998 log.cc:1196] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/5cd4bb8bc4054d8bbe0b4e0ba1d970b0
I20260806 09:11:23.682003 13998 ts_tablet_manager.cc:1939] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P 4e97f231174d4f2f92a35db0df1fad6c: Deleting consensus metadata
I20260806 09:11:23.682083 13410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:23.683193 13137 catalog_manager.cc:4841] TS 4e97f231174d4f2f92a35db0df1fad6c (127.3.69.3:36007): tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 (table test-table [id=ef46df391667410dbeabf608a38bcef3]) successfully deleted
I20260806 09:11:23.685161 14000 ts_tablet_manager.cc:1918] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:23.685349 14000 log.cc:1196] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/5cd4bb8bc4054d8bbe0b4e0ba1d970b0
I20260806 09:11:23.685794 14000 ts_tablet_manager.cc:1939] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P e4d35582e97a4bfea3165b8f33a3606b: Deleting consensus metadata
I20260806 09:11:23.686844 13138 catalog_manager.cc:4841] TS e4d35582e97a4bfea3165b8f33a3606b (127.3.69.2:40105): tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 (table test-table [id=ef46df391667410dbeabf608a38bcef3]) successfully deleted
I20260806 09:11:23.683454 13999 ts_tablet_manager.cc:1905] T 5cd4bb8bc4054d8bbe0b4e0ba1d970b0 P c59aeb9db55843f8af2e0a3197e5ea96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:23.808884 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 13214
I20260806 09:11:23.844231 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 13434
I20260806 09:11:23.844540 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.844056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56864)
0806 09:11:23.844247 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.844356 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 09:11:23.844887 13137 catalog_manager.cc:4584] TS c59aeb9db55843f8af2e0a3197e5ea96 (127.3.69.1:45171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cd4bb8bc4054d8bbe0b4e0ba1d970b0: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
I20260806 09:11:23.859608 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.845107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.3.69.2:40105)
0806 09:11:23.845299 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:23.845306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:23.845426 (+ 120us) server_negotiation.cc:407] Connection header received
0806 09:11:23.845658 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:23.845662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:23.845735 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:23.845852 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:23.847336 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.848575 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.849829 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:23.850565 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:23.850706 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:23.853382 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:23.853396 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:23.853412 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:23.853452 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:23.856765 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:23.857463 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:23.857470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:23.857472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:23.857631 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:23.858022 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:23.858033 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:23.858036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:23.858369 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:23.858737 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:23.858965 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 09:11:23.859240 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:11:23.860325 13625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:23.905442 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.904953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56866)
0806 09:11:23.905181 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.905278 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:11:23.906555 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.906173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56880)
0806 09:11:23.906364 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.906433 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 09:11:23.975284 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.974726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56888)
0806 09:11:23.974940 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.975054 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:11:23.976302 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.975965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56902)
0806 09:11:23.976121 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.976200 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 09:11:23.980590 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 13434
I20260806 09:11:23.995126 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.994615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44592)
0806 09:11:23.994808 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.994911 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 09:11:23.996256 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:23.995849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:46261 (local address 127.0.0.1:44606)
0806 09:11:23.996036 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:23.996128 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 09:11:24.010699 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 13652
I20260806 09:11:24.025228 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:24.011691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.3.69.3:36007)
0806 09:11:24.011914 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:24.011920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:24.011996 (+ 76us) server_negotiation.cc:407] Connection header received
0806 09:11:24.012227 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:24.012232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:24.012314 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:24.012428 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:24.013903 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:24.014666 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:24.016100 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:24.016513 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:24.016663 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:24.019290 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:24.019307 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:24.019324 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:24.019358 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:24.022336 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:24.023015 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:24.023024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:24.023030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:24.023244 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:24.023652 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:24.023657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:24.023661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:24.024033 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:24.024354 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:24.024658 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 09:11:24.024877 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:11:24.025903 13825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:24.077509 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:24.077003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56906)
0806 09:11:24.077245 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:24.077334 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 09:11:24.078559 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:24.078215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56922)
0806 09:11:24.078371 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:24.078449 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 09:11:24.120039 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 13652
I20260806 09:11:24.141177 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 12701
I20260806 09:11:24.142313 12952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:24.249964 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:24.249429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56936)
0806 09:11:24.249695 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:24.249793 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 09:11:24.251293 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:24.250847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:45171 (local address 127.0.0.1:56940)
0806 09:11:24.251052 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:24.251176 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 09:11:24.263604 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 12701
I20260806 09:11:24.289593 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 12859
I20260806 09:11:24.290784 13203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:24.413071 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 12859
2026-08-06T09:11:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10121 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:11:24 dist-test-slave-w4v5 krb5kdc[14022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:11:24 dist-test-slave-w4v5 krb5kdc[14022](info): set up 2 sockets
Aug 06 09:11:24 dist-test-slave-w4v5 krb5kdc[14022](info): commencing operation
krb5kdc: starting...
W20260806 09:11:25.736604 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:11:25 dist-test-slave-w4v5 krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:11:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.62.keytab.
Entry for principal oryx/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.62.keytab.
I20260806 09:11:25.988680 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39659
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.62.keytab
--principal=oryx/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:39659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:26.324059 14041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:26.324342 14041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:26.324522 14041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:26.331678 14041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:26.331781 14041 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:11:26.331871 14041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:26.331981 14041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:26.332070 14041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:26.332155 14041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:26.341672 14041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:39659
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.62.keytab
--principal=oryx/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:26.342921 14041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:26.344259 14041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:26.355942 14050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:26.355851 14053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:26.355906 14051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:26.357511 14041 server_base.cc:1034] running on GCE node
I20260806 09:11:26.358225 14041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:26.359683 14041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:26.360951 14041 hybrid_clock.cc:648] HybridClock initialized: now 1786007486360890 us; error 34 us; skew 500 ppm
I20260806 09:11:26.361253 14041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:26 dist-test-slave-w4v5 krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007486, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:26.368391 14041 init.cc:377] Logged in from keytab as oryx/127.3.69.62@KRBTEST.COM (short username oryx)
I20260806 09:11:26.369954 14041 webserver.cc:460] Webserver started at http://127.3.69.62:38679/ using document root <none> and password file <none>
I20260806 09:11:26.370666 14041 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:26.370757 14041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:26.371044 14041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:26.374392 14041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "0d649e5b6c584cf9a8060d2cece13db2"
format_stamp: "Formatted at 2026-08-06 09:11:26 on dist-test-slave-w4v5"
server_key: "61eee2fc01f549dfb2c3a64d95bb1e32"
server_key_iv: "879d837df50ea5049d2fc9874940ce60"
server_key_version: "encryptionkey@0"
I20260806 09:11:26.375316 14041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "0d649e5b6c584cf9a8060d2cece13db2"
format_stamp: "Formatted at 2026-08-06 09:11:26 on dist-test-slave-w4v5"
server_key: "61eee2fc01f549dfb2c3a64d95bb1e32"
server_key_iv: "879d837df50ea5049d2fc9874940ce60"
server_key_version: "encryptionkey@0"
I20260806 09:11:26.380496 14041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:11:26.384155 14062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:26.385418 14041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:11:26.385594 14041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "0d649e5b6c584cf9a8060d2cece13db2"
format_stamp: "Formatted at 2026-08-06 09:11:26 on dist-test-slave-w4v5"
server_key: "61eee2fc01f549dfb2c3a64d95bb1e32"
server_key_iv: "879d837df50ea5049d2fc9874940ce60"
server_key_version: "encryptionkey@0"
I20260806 09:11:26.385852 14041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:26.409597 14041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:26.413815 14041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:26.414095 14041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:26.436861 14041 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:39659
I20260806 09:11:26.436903 14139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:39659 every 8 connection(s)
I20260806 09:11:26.438237 14041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:26.442039 14140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:26.443236 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 14041
I20260806 09:11:26.443464 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:26.444321 3348 external_mini_cluster.cc:1428] Setting key 4bc4c8d62bdf63f598e98c67bf913418
I20260806 09:11:26.456845 14140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: Bootstrap starting.
Aug 06 09:11:26 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:26.460570 14140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:26.461625 14140 log.cc:824] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:26.464572 14140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: No bootstrap required, opened a new log
I20260806 09:11:26.467844 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:26.446475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.3.69.62:39659)
0806 09:11:26.447342 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:26.447353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:26.447394 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:11:26.448974 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:26.448999 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:26.449387 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:26.449715 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:26.451547 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:26.452809 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:26.454580 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:26.455272 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:26.455396 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:26.460194 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:26.460203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:26.460217 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:26.460252 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:26.463780 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:26.464445 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:26.464455 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:26.464467 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:26.464637 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:26.465177 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:26.465181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:26.465183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:26.465669 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:26.465897 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:26.466396 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 09:11:26.466757 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":415,"threads_started":1}
I20260806 09:11:26.472658 14140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d649e5b6c584cf9a8060d2cece13db2" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39659 } }
I20260806 09:11:26.473018 14140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:26.473124 14140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d649e5b6c584cf9a8060d2cece13db2, State: Initialized, Role: FOLLOWER
I20260806 09:11:26.473655 14140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d649e5b6c584cf9a8060d2cece13db2" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39659 } }
I20260806 09:11:26.473903 14140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:26.474045 14140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:26.474225 14140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:26.476449 14140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d649e5b6c584cf9a8060d2cece13db2" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39659 } }
I20260806 09:11:26.476950 14140 leader_election.cc:302] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d649e5b6c584cf9a8060d2cece13db2; no voters:
I20260806 09:11:26.477514 14140 leader_election.cc:288] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:26.477643 14147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:26.478432 14147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [term 1 LEADER]: Becoming Leader. State: Replica: 0d649e5b6c584cf9a8060d2cece13db2, State: Running, Role: LEADER
I20260806 09:11:26.479007 14147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d649e5b6c584cf9a8060d2cece13db2" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39659 } }
I20260806 09:11:26.479805 14140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:26.482200 14150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d649e5b6c584cf9a8060d2cece13db2. Latest consensus state: current_term: 1 leader_uuid: "0d649e5b6c584cf9a8060d2cece13db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d649e5b6c584cf9a8060d2cece13db2" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39659 } } }
I20260806 09:11:26.482597 14150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:26.483582 14149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d649e5b6c584cf9a8060d2cece13db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d649e5b6c584cf9a8060d2cece13db2" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39659 } } }
I20260806 09:11:26.484556 14149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:26.486109 14153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:26.491787 14153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:26.500519 14153 catalog_manager.cc:1347] Generated new cluster ID: 1a12bb201f3b487185d3d726c5a2d46c
I20260806 09:11:26.500633 14153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:26.511575 14153 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:26.513329 14153 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:26.520223 14153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: Generated new TSK 0
I20260806 09:11:26.521179 14153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab.
Entry for principal oryx/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab.
I20260806 09:11:26.585031 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39659
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab
--principal=oryx/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:26.939983 14176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:26.940290 14176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:26.940444 14176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:26.947790 14176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:26.947902 14176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:26.948056 14176 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:11:26.948196 14176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:26.958173 14176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab
--principal=oryx/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:39659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:26.959607 14176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:26.961577 14176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:26.978588 14183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:26.978648 14187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:26.979241 14184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:26.979377 14176 server_base.cc:1034] running on GCE node
I20260806 09:11:26.980161 14176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:26.981315 14176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:26.982601 14176 hybrid_clock.cc:648] HybridClock initialized: now 1786007486982530 us; error 36 us; skew 500 ppm
I20260806 09:11:26.982963 14176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:26 dist-test-slave-w4v5 krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007486, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:26.986330 14176 init.cc:377] Logged in from keytab as oryx/127.3.69.1@KRBTEST.COM (short username oryx)
I20260806 09:11:26.987906 14176 webserver.cc:460] Webserver started at http://127.3.69.1:44103/ using document root <none> and password file <none>
I20260806 09:11:26.988642 14176 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:26.988780 14176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:26.989136 14176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:26.992194 14176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e"
format_stamp: "Formatted at 2026-08-06 09:11:26 on dist-test-slave-w4v5"
server_key: "42d7051f4091fb93ea009bd2140d90fa"
server_key_iv: "172cb109ed2d536a78a4f6cdfbdfcd75"
server_key_version: "encryptionkey@0"
I20260806 09:11:26.992996 14176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e"
format_stamp: "Formatted at 2026-08-06 09:11:26 on dist-test-slave-w4v5"
server_key: "42d7051f4091fb93ea009bd2140d90fa"
server_key_iv: "172cb109ed2d536a78a4f6cdfbdfcd75"
server_key_version: "encryptionkey@0"
I20260806 09:11:26.998282 14176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 09:11:27.001981 14198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:27.003482 14176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:27.003722 14176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e"
format_stamp: "Formatted at 2026-08-06 09:11:26 on dist-test-slave-w4v5"
server_key: "42d7051f4091fb93ea009bd2140d90fa"
server_key_iv: "172cb109ed2d536a78a4f6cdfbdfcd75"
server_key_version: "encryptionkey@0"
I20260806 09:11:27.003963 14176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:27.026796 14176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:27.031222 14176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:27.031502 14176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:27.034216 14176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:27.034337 14176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:27.034489 14176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:27.034583 14176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:11:27 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007486, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:27.059909 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:27.035058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:38473 (local address 127.3.69.62:39659)
0806 09:11:27.035575 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:27.035581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:27.036906 (+ 1325us) server_negotiation.cc:407] Connection header received
0806 09:11:27.040141 (+ 3235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:27.040147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:27.040276 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:27.040412 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:27.042770 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.044136 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.046547 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.047408 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.047588 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:27.052890 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:27.052895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:27.052898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:27.052931 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:27.055646 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:27.056547 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:27.056554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:27.056561 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:27.056744 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:27.057429 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:27.057435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:27.057438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:27.057851 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:27.058047 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:27.059102 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0806 09:11:27.059395 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":256,"threads_started":1}
I20260806 09:11:27.060853 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:27.035735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39659 (local address 127.3.69.1:38473)
0806 09:11:27.036711 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:27.036772 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:27.039752 (+ 2980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:27.040725 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:27.040740 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:27.041600 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:27.042499 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:27.042524 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.044361 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.044369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:27.046337 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:27.046380 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.047765 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.047776 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:27.048030 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:27.048897 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:27.048926 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:27.052581 (+ 3655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:27.055799 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:27.055808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:27.055825 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:27.056384 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:27.056991 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:27.056996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:27.056999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:27.057278 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:27.058030 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:27.058042 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:27.058505 (+ 463us) client_negotiation.cc:769] Sending connection context
0806 09:11:27.058928 (+ 423us) client_negotiation.cc:240] Negotiation successful
0806 09:11:27.059568 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":373,"threads_started":1}
I20260806 09:11:27.080917 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:27.066844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:41139 (local address 127.3.69.62:39659)
0806 09:11:27.067099 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:27.067108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:27.067743 (+ 635us) server_negotiation.cc:407] Connection header received
0806 09:11:27.067891 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:27.067900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:27.068000 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:27.068144 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:27.069019 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.070338 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.072606 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.073405 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.073551 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:27.075503 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:27.075508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:27.075511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:27.075540 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:27.078077 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:27.078728 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:27.078733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:27.078738 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:27.078862 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:27.079323 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:27.079328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:27.079331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:27.079784 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:27.079942 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:27.080347 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 09:11:27.080565 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:11:27.081028 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:27.066824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39659 (local address 127.3.69.1:41139)
0806 09:11:27.067645 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:27.067662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:27.067774 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:27.068155 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:27.068158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:27.068598 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:27.068880 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:27.068889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.070525 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.070533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:27.072346 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:27.072358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.073661 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.073668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:27.073822 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:27.074499 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:27.074515 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:27.075307 (+ 792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:27.078217 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:27.078222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:27.078226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:27.078599 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:27.078985 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:27.078990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:27.078993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:27.079202 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:27.079902 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:27.079909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:27.080093 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 09:11:27.080235 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 09:11:27.080596 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":512,"threads_started":1}
I20260806 09:11:27.098708 14176 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:42655
I20260806 09:11:27.098745 14364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:42655 every 8 connection(s)
I20260806 09:11:27.100062 14176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:27.102280 14366 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39659
I20260806 09:11:27.102491 14366 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:27.103186 14366 heartbeater.cc:511] Master 127.3.69.62:39659 requested a full tablet report, sending...
I20260806 09:11:27.105171 14082 ts_manager.cc:194] Registered new tserver with Master: d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655)
I20260806 09:11:27.106614 14082 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:38473
I20260806 09:11:27.108207 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 14176
I20260806 09:11:27.108413 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:11:27.109236 3348 external_mini_cluster.cc:1428] Setting key 68fd2f356abbd1b9c02ab1f83e27bad0
WARNING: no policy specified for oryx/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab.
Entry for principal oryx/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab.
I20260806 09:11:27.162493 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39659
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab
--principal=oryx/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:27.494452 14374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:27.494727 14374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:27.494824 14374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:27.502110 14374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:27.502211 14374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:27.502377 14374 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:11:27.502514 14374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:27.512293 14374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab
--principal=oryx/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:39659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:27.513513 14374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:27.514814 14374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:27.527921 14381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:27.528024 14382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:27.527982 14384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:27.529201 14374 server_base.cc:1034] running on GCE node
I20260806 09:11:27.529727 14374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:27.530551 14374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:27.531787 14374 hybrid_clock.cc:648] HybridClock initialized: now 1786007487531716 us; error 49 us; skew 500 ppm
I20260806 09:11:27.532075 14374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:27 dist-test-slave-w4v5 krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007487, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:27.535212 14374 init.cc:377] Logged in from keytab as oryx/127.3.69.2@KRBTEST.COM (short username oryx)
I20260806 09:11:27.536631 14374 webserver.cc:460] Webserver started at http://127.3.69.2:33105/ using document root <none> and password file <none>
I20260806 09:11:27.537295 14374 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:27.537425 14374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:27.537753 14374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:27.540570 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e"
format_stamp: "Formatted at 2026-08-06 09:11:27 on dist-test-slave-w4v5"
server_key: "28d3aca629bd278037fe5770408056d8"
server_key_iv: "e1a9dac090308035f56960f5798f92ce"
server_key_version: "encryptionkey@0"
I20260806 09:11:27.541368 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e"
format_stamp: "Formatted at 2026-08-06 09:11:27 on dist-test-slave-w4v5"
server_key: "28d3aca629bd278037fe5770408056d8"
server_key_iv: "e1a9dac090308035f56960f5798f92ce"
server_key_version: "encryptionkey@0"
I20260806 09:11:27.546229 14374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 09:11:27.549665 14395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:27.550925 14374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:11:27.551141 14374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e"
format_stamp: "Formatted at 2026-08-06 09:11:27 on dist-test-slave-w4v5"
server_key: "28d3aca629bd278037fe5770408056d8"
server_key_iv: "e1a9dac090308035f56960f5798f92ce"
server_key_version: "encryptionkey@0"
I20260806 09:11:27.551362 14374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:27.571722 14374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:27.575829 14374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:27.576102 14374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:27.578878 14374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:27.578994 14374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:27.579155 14374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:27.579272 14374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:11:27 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007487, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:27.603924 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:27.579383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:36149 (local address 127.3.69.62:39659)
0806 09:11:27.579596 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:27.579605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:27.581127 (+ 1522us) server_negotiation.cc:407] Connection header received
0806 09:11:27.583374 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:27.583380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:27.583481 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:27.583608 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:27.585758 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.587005 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.589721 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.590593 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.590772 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:27.595798 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:27.595808 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:27.595816 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:27.595871 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:27.599979 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:27.600989 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:27.600998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:27.601004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:27.601199 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:27.601646 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:27.601653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:27.601659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:27.602079 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:27.602266 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:27.603328 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0806 09:11:27.603563 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:11:27.604992 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:27.579977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39659 (local address 127.3.69.2:36149)
0806 09:11:27.580947 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:27.580990 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:27.583001 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:27.583839 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:27.583847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:27.584673 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:27.585539 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:27.585564 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.587260 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.587265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:27.589496 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:27.589509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.590901 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.590913 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:27.591292 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:27.592052 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:27.592100 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:27.595546 (+ 3446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:27.600144 (+ 4598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:27.600156 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:27.600175 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:27.600823 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:27.601333 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:27.601336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:27.601339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:27.601521 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:27.602259 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:27.602272 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:27.602750 (+ 478us) client_negotiation.cc:769] Sending connection context
0806 09:11:27.603207 (+ 457us) client_negotiation.cc:240] Negotiation successful
0806 09:11:27.603803 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":393,"threads_started":1}
I20260806 09:11:27.634524 14374 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:42335
I20260806 09:11:27.634550 14568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:42335 every 8 connection(s)
I20260806 09:11:27.636005 14374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:27.636463 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 14374
I20260806 09:11:27.636710 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:11:27.637312 3348 external_mini_cluster.cc:1428] Setting key 02f9868c03970daa1dd47d5a6aaa7cf2
I20260806 09:11:27.639122 14569 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39659
I20260806 09:11:27.639133 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:27.611441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39659 (local address 127.3.69.2:44021)
0806 09:11:27.623609 (+ 12168us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:27.623637 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:27.623871 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:27.624329 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:27.624337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:27.625283 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:27.625911 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:27.625932 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.627053 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.627060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:27.629352 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:27.629371 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.630135 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.630144 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:27.630386 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:27.631172 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:27.631215 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:27.632545 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:27.635806 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:27.635811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:27.635815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:27.636344 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:27.636843 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:27.636848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:27.636851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:27.637014 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:27.637827 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:27.637832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:27.638082 (+ 250us) client_negotiation.cc:769] Sending connection context
0806 09:11:27.638217 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 09:11:27.638548 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11944,"thread_start_us":11783,"threads_started":1}
I20260806 09:11:27.639362 14569 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:27.639941 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:27.611331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:44021 (local address 127.3.69.62:39659)
0806 09:11:27.611518 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:27.611525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:27.623742 (+ 12217us) server_negotiation.cc:407] Connection header received
0806 09:11:27.624020 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:27.624027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:27.624143 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:27.624261 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:27.626160 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.626896 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.629518 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:27.629926 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:27.630026 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:27.632742 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:27.632746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:27.632750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:27.632782 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:27.635677 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:27.636505 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:27.636511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:27.636514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:27.636673 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:27.637188 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:27.637192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:27.637195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:27.637664 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:27.639174 (+ 1510us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:27.639333 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 09:11:27.639563 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 09:11:27.640096 14569 heartbeater.cc:511] Master 127.3.69.62:39659 requested a full tablet report, sending...
I20260806 09:11:27.642504 14080 ts_manager.cc:194] Registered new tserver with Master: 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2:42335)
I20260806 09:11:27.643697 14080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:36149
WARNING: no policy specified for oryx/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab.
Entry for principal oryx/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab.
I20260806 09:11:27.694489 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39659
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab
--principal=oryx/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:28.028525 14574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:28.028802 14574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:28.028967 14574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:28.036281 14574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:28.036391 14574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:28.036531 14574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:11:28.036643 14574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:28.046893 14574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab
--principal=oryx/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.69.62:39659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:28.048193 14574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:28.049579 14574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:28.063720 14585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:28.063836 14582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:28.063876 14583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:28.064445 14574 server_base.cc:1034] running on GCE node
I20260806 09:11:28.065079 14574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:28.065937 14574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:28.067210 14574 hybrid_clock.cc:648] HybridClock initialized: now 1786007488067133 us; error 43 us; skew 500 ppm
I20260806 09:11:28.067509 14574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:28 dist-test-slave-w4v5 krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:28.070578 14574 init.cc:377] Logged in from keytab as oryx/127.3.69.3@KRBTEST.COM (short username oryx)
I20260806 09:11:28.071961 14574 webserver.cc:460] Webserver started at http://127.3.69.3:43649/ using document root <none> and password file <none>
I20260806 09:11:28.072678 14574 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:28.072816 14574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:28.073158 14574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:28.076095 14574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "afb87ebed0274353a9d4a27acc4fc0b7"
format_stamp: "Formatted at 2026-08-06 09:11:28 on dist-test-slave-w4v5"
server_key: "a2cfe731aa686b345bdd248f3d3e9cfa"
server_key_iv: "bb20e8c57f919985cbf5779266a90bda"
server_key_version: "encryptionkey@0"
I20260806 09:11:28.076875 14574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "afb87ebed0274353a9d4a27acc4fc0b7"
format_stamp: "Formatted at 2026-08-06 09:11:28 on dist-test-slave-w4v5"
server_key: "a2cfe731aa686b345bdd248f3d3e9cfa"
server_key_iv: "bb20e8c57f919985cbf5779266a90bda"
server_key_version: "encryptionkey@0"
I20260806 09:11:28.081964 14574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:11:28.085304 14597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:28.086603 14574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 09:11:28.086774 14574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "afb87ebed0274353a9d4a27acc4fc0b7"
format_stamp: "Formatted at 2026-08-06 09:11:28 on dist-test-slave-w4v5"
server_key: "a2cfe731aa686b345bdd248f3d3e9cfa"
server_key_iv: "bb20e8c57f919985cbf5779266a90bda"
server_key_version: "encryptionkey@0"
I20260806 09:11:28.086997 14574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:28.109571 14366 heartbeater.cc:503] Master 127.3.69.62:39659 was elected leader, sending a full tablet report...
I20260806 09:11:28.114013 14574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:28.117913 14574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:28.118201 14574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:28.120944 14574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:28.121057 14574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:28.121284 14574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:28.121513 14574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:11:28 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:28.146180 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.121297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:60579 (local address 127.3.69.62:39659)
0806 09:11:28.121462 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.121467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.123168 (+ 1701us) server_negotiation.cc:407] Connection header received
0806 09:11:28.125233 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.125239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.125328 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.125449 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:28.127762 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.128893 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.131804 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.132386 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.132523 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.138126 (+ 5603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:28.138132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:28.138135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:28.138169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:28.141987 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:28.143064 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:28.143070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:28.143092 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:28.143293 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:28.143906 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:28.143915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:28.143921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:28.144382 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:28.144641 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.145339 (+ 698us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.145657 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 09:11:28.146680 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.121977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39659 (local address 127.3.69.3:60579)
0806 09:11:28.122958 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:28.123011 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:28.124939 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:28.125689 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:28.125707 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:28.126552 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:28.127549 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.127583 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.129064 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.129069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.131622 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.131636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.132847 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.132857 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.133154 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.133972 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:28.133997 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:28.137852 (+ 3855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:28.142165 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:28.142178 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:28.142197 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:28.142899 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:28.143423 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:28.143426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:28.143429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:28.143760 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:28.144590 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:28.144596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:28.144932 (+ 336us) client_negotiation.cc:769] Sending connection context
0806 09:11:28.145161 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 09:11:28.145529 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":443,"threads_started":1}
I20260806 09:11:28.167550 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.154394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:50727 (local address 127.3.69.62:39659)
0806 09:11:28.154630 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.154636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.154884 (+ 248us) server_negotiation.cc:407] Connection header received
0806 09:11:28.155191 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.155210 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.155289 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.155400 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:28.156373 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.157075 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.158400 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.158812 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.158951 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.161160 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:28.161164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:28.161167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:28.161200 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:28.164438 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:28.165398 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:28.165401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:28.165404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:28.165522 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:28.165973 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:28.165976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:28.165978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:28.166226 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:28.166344 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.166908 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.167070 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260806 09:11:28.167989 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.153950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39659 (local address 127.3.69.3:50727)
0806 09:11:28.154768 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:28.154788 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:28.154961 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:28.155425 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:28.155430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:28.155939 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:28.156229 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.156238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.157216 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.157225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.158249 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.158260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.159014 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.159022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.159264 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.159959 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:28.159975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:28.160915 (+ 940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:28.164578 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:28.164587 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:28.164594 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:28.165268 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:28.165638 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:28.165645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:28.165651 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:28.165851 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:28.166343 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:28.166351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:28.166621 (+ 270us) client_negotiation.cc:769] Sending connection context
0806 09:11:28.166813 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 09:11:28.167278 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":692,"threads_started":1}
I20260806 09:11:28.181433 14574 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:41719
I20260806 09:11:28.181478 14766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:41719 every 8 connection(s)
I20260806 09:11:28.182668 14574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:28.184708 14767 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39659
I20260806 09:11:28.184899 14767 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:28.185490 14767 heartbeater.cc:511] Master 127.3.69.62:39659 requested a full tablet report, sending...
I20260806 09:11:28.186919 14080 ts_manager.cc:194] Registered new tserver with Master: afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:28.187860 14080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:60579
I20260806 09:11:28.190013 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 14574
I20260806 09:11:28.190196 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:11:28.190723 3348 external_mini_cluster.cc:1428] Setting key 88e5cd1b8042411e71f70ea51714b6d0
I20260806 09:11:28.193747 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:11:28 dist-test-slave-w4v5 krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007488, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:11:28 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:28 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:11:28.250341 14212 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:28.239920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.3.69.62:39659)
0806 09:11:28.240095 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.240100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.240389 (+ 289us) server_negotiation.cc:407] Connection header received
0806 09:11:28.240548 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.240552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.240620 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.240714 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:28.242046 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.242762 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.244131 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.244520 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.244616 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.250071 (+ 5455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45678: BlockingRecv error: recv got EOF from 127.0.0.1:45678 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 09:11:28 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007488, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:28.267342 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.253526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.3.69.62:39659)
0806 09:11:28.253734 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.253743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.254000 (+ 257us) server_negotiation.cc:407] Connection header received
0806 09:11:28.254191 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.254207 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.254303 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.254414 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:28.255723 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.256493 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.257694 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.258116 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.258277 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.261892 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:28.261905 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:28.261909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:28.261941 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:28.264752 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:28.265372 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:28.265379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:28.265382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:28.265519 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:28.265931 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:28.265936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:28.265940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:28.266226 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:28.266512 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.266785 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.266995 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:11:28.271263 14080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:11:28.278262 14080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:28.306984 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.291180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41719 (local address 127.0.0.1:37494)
0806 09:11:28.293374 (+ 2194us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:28.293396 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:28.293512 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:28.296583 (+ 3071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:28.296595 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:28.296640 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:28.297209 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.297227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.300323 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.300332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.303552 (+ 3220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.303574 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.305888 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.305894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.306123 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.306200 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 09:11:28.306394 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 09:11:28.306604 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2072,"thread_start_us":439,"threads_started":1}
I20260806 09:11:28.308558 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.291865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.3.69.3:41719)
0806 09:11:28.295768 (+ 3903us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.295781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.295824 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:11:28.295945 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.295957 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.296286 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.297489 (+ 1203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:28.297898 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.300107 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.303836 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.305694 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.305831 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.306088 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.307003 (+ 915us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.307209 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3667,"thread_start_us":3462,"threads_started":1}
I20260806 09:11:28.313675 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.292815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:42335 (local address 127.0.0.1:53656)
0806 09:11:28.306519 (+ 13704us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:28.306536 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:28.306643 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:28.307460 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:28.307465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:28.307494 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:28.307937 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.307949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.309937 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.309943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.311519 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.311532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.312871 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.312879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.313043 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.313075 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:11:28.313183 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 09:11:28.313314 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13582,"mutex_wait_us":3,"thread_start_us":578,"threads_started":1}
I20260806 09:11:28.313704 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.293211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.3.69.2:42335)
0806 09:11:28.305518 (+ 12307us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.305528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.306773 (+ 1245us) server_negotiation.cc:407] Connection header received
0806 09:11:28.306877 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.306885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.307245 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.308206 (+ 961us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:28.308384 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.309787 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.311684 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.312655 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.312791 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.312997 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.313236 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.313404 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12131,"thread_start_us":12308,"threads_started":1}
I20260806 09:11:28.312904 14667 tablet_service.cc:1467] Processing CreateTablet for tablet 057a9f16fcdd42d1abee1ed7caad0d20 (DEFAULT_TABLE table=test-table [id=89ce94948fe443ddb22605927676d456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:28.315146 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057a9f16fcdd42d1abee1ed7caad0d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.315591 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.291068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42655 (local address 127.0.0.1:56200)
0806 09:11:28.291952 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:28.291987 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:28.292158 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:28.307658 (+ 15500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:28.307665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:28.307697 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:28.308299 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.308313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.310095 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.310100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.312286 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.312301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.314833 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.314838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.314951 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.314974 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 09:11:28.315084 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 09:11:28.315252 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":15129,"threads_started":1}
I20260806 09:11:28.316294 14792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.291293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.3.69.1:42655)
0806 09:11:28.306971 (+ 15678us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.306979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.307005 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:11:28.307184 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.307192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.307416 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.308530 (+ 1114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:28.308763 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.309935 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.312460 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.313943 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.314505 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.314724 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.315831 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.316012 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15525,"thread_start_us":15379,"threads_started":1}
I20260806 09:11:28.317602 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 057a9f16fcdd42d1abee1ed7caad0d20 (DEFAULT_TABLE table=test-table [id=89ce94948fe443ddb22605927676d456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:28.318994 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057a9f16fcdd42d1abee1ed7caad0d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.320097 14265 tablet_service.cc:1467] Processing CreateTablet for tablet 057a9f16fcdd42d1abee1ed7caad0d20 (DEFAULT_TABLE table=test-table [id=89ce94948fe443ddb22605927676d456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:28.321796 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 057a9f16fcdd42d1abee1ed7caad0d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.327029 14799 tablet_bootstrap.cc:492] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Bootstrap starting.
I20260806 09:11:28.330154 14799 tablet_bootstrap.cc:654] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.331418 14799 log.cc:824] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:28.333868 14799 tablet_bootstrap.cc:492] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: No bootstrap required, opened a new log
I20260806 09:11:28.334226 14799 ts_tablet_manager.cc:1397] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20260806 09:11:28.334694 14802 tablet_bootstrap.cc:492] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Bootstrap starting.
I20260806 09:11:28.336097 14804 tablet_bootstrap.cc:492] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Bootstrap starting.
I20260806 09:11:28.338151 14802 tablet_bootstrap.cc:654] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.338883 14804 tablet_bootstrap.cc:654] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.339391 14802 log.cc:824] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:28.340013 14804 log.cc:824] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:28.342682 14804 tablet_bootstrap.cc:492] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: No bootstrap required, opened a new log
I20260806 09:11:28.342793 14802 tablet_bootstrap.cc:492] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: No bootstrap required, opened a new log
I20260806 09:11:28.342976 14804 ts_tablet_manager.cc:1397] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 09:11:28.343046 14802 ts_tablet_manager.cc:1397] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 09:11:28.345141 14799 raft_consensus.cc:348] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.345568 14799 raft_consensus.cc:374] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.345666 14799 raft_consensus.cc:729] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb87ebed0274353a9d4a27acc4fc0b7, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.346269 14799 consensus_queue.cc:260] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.347391 14767 heartbeater.cc:503] Master 127.3.69.62:39659 was elected leader, sending a full tablet report...
I20260806 09:11:28.350247 14799 ts_tablet_manager.cc:1428] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260806 09:11:28.352051 14804 raft_consensus.cc:348] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.352510 14804 raft_consensus.cc:374] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.352644 14804 raft_consensus.cc:729] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a6bd8be7fa4c1482af1ded80bbcf9e, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.353451 14804 consensus_queue.cc:260] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.353593 14802 raft_consensus.cc:348] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.354019 14802 raft_consensus.cc:374] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.354126 14802 raft_consensus.cc:729] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.354671 14802 consensus_queue.cc:260] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.355736 14569 heartbeater.cc:503] Master 127.3.69.62:39659 was elected leader, sending a full tablet report...
I20260806 09:11:28.356473 14804 ts_tablet_manager.cc:1428] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260806 09:11:28.357451 14802 ts_tablet_manager.cc:1428] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
W20260806 09:11:28.389276 14570 tablet.cc:2369] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:11:28.434427 14768 tablet.cc:2369] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:28.591776 14809 raft_consensus.cc:482] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:28.592183 14809 raft_consensus.cc:504] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.593935 14809 leader_election.cc:288] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:28.600680 14792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.594289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:44563 (local address 127.3.69.1:42655)
0806 09:11:28.594550 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.594557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.595289 (+ 732us) server_negotiation.cc:407] Connection header received
0806 09:11:28.595466 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.595472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.595544 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.595642 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:28.596285 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.597040 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.598843 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.599675 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.599829 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.599948 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.600289 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.600411 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:11:28.600718 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.594768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:54587 (local address 127.3.69.3:41719)
0806 09:11:28.594963 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.594969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.595139 (+ 170us) server_negotiation.cc:407] Connection header received
0806 09:11:28.595420 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.595426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.595500 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.595604 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:28.596315 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.597005 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.598842 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.599686 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.599836 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.599963 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.600307 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.600433 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:11:28.600916 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.594236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41719 (local address 127.3.69.2:54587)
0806 09:11:28.594988 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:28.595016 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:28.595273 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:28.595619 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:28.595623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:28.595657 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:28.596147 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.596161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.597138 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.597146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.598677 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.598689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.599964 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.599971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.600138 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.600162 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:11:28.600278 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 09:11:28.600408 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":9,"thread_start_us":781,"threads_started":1}
I20260806 09:11:28.600960 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.594301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42655 (local address 127.3.69.2:44563)
0806 09:11:28.595163 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:28.595183 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:28.595313 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:28.595656 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:28.595660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:28.595684 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:28.596124 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.596136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.597181 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.597186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.598690 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.598702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.599938 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.599947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.600118 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.600144 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 09:11:28.600251 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 09:11:28.600379 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":25,"thread_start_us":454,"threads_started":1}
I20260806 09:11:28.602012 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057a9f16fcdd42d1abee1ed7caad0d20" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" is_pre_election: true
I20260806 09:11:28.602016 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057a9f16fcdd42d1abee1ed7caad0d20" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" is_pre_election: true
I20260806 09:11:28.602509 14704 raft_consensus.cc:2454] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 0.
I20260806 09:11:28.602563 14296 raft_consensus.cc:2454] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 0.
W20260806 09:11:28.602651 14369 tablet.cc:2369] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:11:28.603256 14399 leader_election.cc:302] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bca3e5ba3d4d7db69f7d69002b1b2e, afb87ebed0274353a9d4a27acc4fc0b7; no voters:
I20260806 09:11:28.603649 14809 raft_consensus.cc:2781] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:28.603812 14809 raft_consensus.cc:482] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:28.603929 14809 raft_consensus.cc:3037] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.606295 14809 raft_consensus.cc:504] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.606875 14809 leader_election.cc:288] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 election: Requested vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:28.607227 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057a9f16fcdd42d1abee1ed7caad0d20" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e"
I20260806 09:11:28.607398 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "057a9f16fcdd42d1abee1ed7caad0d20" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb87ebed0274353a9d4a27acc4fc0b7"
I20260806 09:11:28.607465 14296 raft_consensus.cc:3037] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.607651 14704 raft_consensus.cc:3037] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.610002 14704 raft_consensus.cc:2454] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 1.
I20260806 09:11:28.610013 14296 raft_consensus.cc:2454] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 1.
I20260806 09:11:28.610391 14398 leader_election.cc:302] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bca3e5ba3d4d7db69f7d69002b1b2e, d6a6bd8be7fa4c1482af1ded80bbcf9e; no voters:
I20260806 09:11:28.610709 14809 raft_consensus.cc:2781] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:28.611212 14809 raft_consensus.cc:686] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 LEADER]: Becoming Leader. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Running, Role: LEADER
I20260806 09:11:28.611676 14809 consensus_queue.cc:237] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } }
I20260806 09:11:28.616532 14082 catalog_manager.cc:5495] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2). New cstate: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } health_report { overall_health: HEALTHY } } }
I20260806 09:11:28.668208 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.662640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.3.69.2:42335)
0806 09:11:28.662884 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.662891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.663038 (+ 147us) server_negotiation.cc:407] Connection header received
0806 09:11:28.663318 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.663323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.663400 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.663525 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:28.663961 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.664792 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.666093 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.666480 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.666580 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.666985 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:28.667575 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:28.667692 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.667827 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.667891 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:11:28.676195 14704 raft_consensus.cc:1261] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Refusing update from remote peer 02bca3e5ba3d4d7db69f7d69002b1b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:28.676263 14296 raft_consensus.cc:1261] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Refusing update from remote peer 02bca3e5ba3d4d7db69f7d69002b1b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:28.677207 14809 consensus_queue.cc:1035] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:28.677502 14813 consensus_queue.cc:1035] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:28.708195 14080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:28.708526 14080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:28.712321 14080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: Sending DeleteTablet for 3 replicas of tablet 057a9f16fcdd42d1abee1ed7caad0d20
I20260806 09:11:28.713587 14265 tablet_service.cc:1514] Processing DeleteTablet for tablet 057a9f16fcdd42d1abee1ed7caad0d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:28 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:56200
I20260806 09:11:28.713860 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet 057a9f16fcdd42d1abee1ed7caad0d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:28 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37494
I20260806 09:11:28.714617 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 057a9f16fcdd42d1abee1ed7caad0d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:28 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:53656
I20260806 09:11:28.714744 14080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:11:28.715466 14080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:28.717666 14839 tablet_replica.cc:330] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: stopping tablet replica
I20260806 09:11:28.718142 14839 raft_consensus.cc:2229] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:28.718652 14839 raft_consensus.cc:2258] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:28.721436 14839 ts_tablet_manager.cc:1905] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:28.723377 14265 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca26e7d3ce14c8fb6eca9a414600f23 (DEFAULT_TABLE table=test-table [id=e395f934ea2042eb9eecfd83df269e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:28.723519 14840 tablet_replica.cc:330] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: stopping tablet replica
I20260806 09:11:28.723961 14840 raft_consensus.cc:2229] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:11:28.724512 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca26e7d3ce14c8fb6eca9a414600f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.724496 14464 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca26e7d3ce14c8fb6eca9a414600f23 (DEFAULT_TABLE table=test-table [id=e395f934ea2042eb9eecfd83df269e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:28.724750 14840 raft_consensus.cc:2258] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:28.725119 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca26e7d3ce14c8fb6eca9a414600f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.725154 14667 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca26e7d3ce14c8fb6eca9a414600f23 (DEFAULT_TABLE table=test-table [id=e395f934ea2042eb9eecfd83df269e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:28.726056 14838 tablet_replica.cc:330] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: stopping tablet replica
I20260806 09:11:28.727563 14838 raft_consensus.cc:2229] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:28.728044 14838 raft_consensus.cc:2258] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:28.728374 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca26e7d3ce14c8fb6eca9a414600f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.731732 14804 tablet_bootstrap.cc:492] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Bootstrap starting.
I20260806 09:11:28.732292 14802 tablet_bootstrap.cc:492] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Bootstrap starting.
I20260806 09:11:28.733451 14804 tablet_bootstrap.cc:654] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.733827 14802 tablet_bootstrap.cc:654] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.734728 14799 tablet_bootstrap.cc:492] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: Bootstrap starting.
I20260806 09:11:28.736649 14799 tablet_bootstrap.cc:654] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.737365 14840 ts_tablet_manager.cc:1905] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:28.740542 14838 ts_tablet_manager.cc:1905] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:28.744983 14839 ts_tablet_manager.cc:1918] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:28.745175 14839 log.cc:1196] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/057a9f16fcdd42d1abee1ed7caad0d20
I20260806 09:11:28.745815 14839 ts_tablet_manager.cc:1939] T 057a9f16fcdd42d1abee1ed7caad0d20 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting consensus metadata
I20260806 09:11:28.748083 14067 catalog_manager.cc:4841] TS afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719): tablet 057a9f16fcdd42d1abee1ed7caad0d20 (table test-table [id=89ce94948fe443ddb22605927676d456]) successfully deleted
I20260806 09:11:28.751488 14799 tablet_bootstrap.cc:492] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: No bootstrap required, opened a new log
I20260806 09:11:28.751652 14799 ts_tablet_manager.cc:1397] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20260806 09:11:28.752032 14838 ts_tablet_manager.cc:1918] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:28.752223 14838 log.cc:1196] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/057a9f16fcdd42d1abee1ed7caad0d20
I20260806 09:11:28.752493 14799 raft_consensus.cc:348] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.752748 14799 raft_consensus.cc:374] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.752862 14838 ts_tablet_manager.cc:1939] T 057a9f16fcdd42d1abee1ed7caad0d20 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting consensus metadata
I20260806 09:11:28.752861 14799 raft_consensus.cc:729] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb87ebed0274353a9d4a27acc4fc0b7, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.753175 14799 consensus_queue.cc:260] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.753799 14799 ts_tablet_manager.cc:1428] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 09:11:28.754674 14840 ts_tablet_manager.cc:1918] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:28.754844 14840 log.cc:1196] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/057a9f16fcdd42d1abee1ed7caad0d20
I20260806 09:11:28.754949 14804 tablet_bootstrap.cc:492] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: No bootstrap required, opened a new log
I20260806 09:11:28.755457 14840 ts_tablet_manager.cc:1939] T 057a9f16fcdd42d1abee1ed7caad0d20 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting consensus metadata
I20260806 09:11:28.755641 14804 ts_tablet_manager.cc:1397] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.001s
I20260806 09:11:28.755760 14065 catalog_manager.cc:4841] TS d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655): tablet 057a9f16fcdd42d1abee1ed7caad0d20 (table test-table [id=89ce94948fe443ddb22605927676d456]) successfully deleted
I20260806 09:11:28.757579 14067 catalog_manager.cc:4841] TS 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2:42335): tablet 057a9f16fcdd42d1abee1ed7caad0d20 (table test-table [id=89ce94948fe443ddb22605927676d456]) successfully deleted
I20260806 09:11:28.758225 14804 raft_consensus.cc:348] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.758631 14804 raft_consensus.cc:374] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.758774 14802 tablet_bootstrap.cc:492] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: No bootstrap required, opened a new log
I20260806 09:11:28.758805 14804 raft_consensus.cc:729] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a6bd8be7fa4c1482af1ded80bbcf9e, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.758931 14802 ts_tablet_manager.cc:1397] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.008s
I20260806 09:11:28.759385 14804 consensus_queue.cc:260] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.759804 14802 raft_consensus.cc:348] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.760036 14802 raft_consensus.cc:374] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.760159 14802 raft_consensus.cc:729] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.760406 14802 consensus_queue.cc:260] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.761004 14802 ts_tablet_manager.cc:1428] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 09:11:28.762696 14804 ts_tablet_manager.cc:1428] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260806 09:11:28.770785 14834 raft_consensus.cc:482] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:28.770995 14834 raft_consensus.cc:504] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.771875 14834 leader_election.cc:288] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:28.772254 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca26e7d3ce14c8fb6eca9a414600f23" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" is_pre_election: true
I20260806 09:11:28.772524 14296 raft_consensus.cc:2454] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 0.
I20260806 09:11:28.772483 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca26e7d3ce14c8fb6eca9a414600f23" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" is_pre_election: true
I20260806 09:11:28.772716 14704 raft_consensus.cc:2454] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 0.
I20260806 09:11:28.773192 14399 leader_election.cc:302] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bca3e5ba3d4d7db69f7d69002b1b2e, afb87ebed0274353a9d4a27acc4fc0b7; no voters:
I20260806 09:11:28.773610 14834 raft_consensus.cc:2781] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:28.773747 14834 raft_consensus.cc:482] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:28.773869 14834 raft_consensus.cc:3037] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.776417 14834 raft_consensus.cc:504] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.777036 14834 leader_election.cc:288] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 election: Requested vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:28.777438 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca26e7d3ce14c8fb6eca9a414600f23" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e"
I20260806 09:11:28.777561 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca26e7d3ce14c8fb6eca9a414600f23" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb87ebed0274353a9d4a27acc4fc0b7"
I20260806 09:11:28.777666 14296 raft_consensus.cc:3037] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.777774 14704 raft_consensus.cc:3037] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.780721 14296 raft_consensus.cc:2454] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 1.
I20260806 09:11:28.780768 14704 raft_consensus.cc:2454] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 1.
I20260806 09:11:28.781227 14398 leader_election.cc:302] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bca3e5ba3d4d7db69f7d69002b1b2e, d6a6bd8be7fa4c1482af1ded80bbcf9e; no voters:
I20260806 09:11:28.781519 14834 raft_consensus.cc:2781] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:28.781683 14834 raft_consensus.cc:686] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 LEADER]: Becoming Leader. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Running, Role: LEADER
I20260806 09:11:28.781988 14834 consensus_queue.cc:237] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.784835 14080 catalog_manager.cc:5495] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2). New cstate: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } health_report { overall_health: UNKNOWN } } }
Aug 06 09:11:28 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007486, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.62@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:28.840608 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.825480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.3.69.62:39659)
0806 09:11:28.826285 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:28.826290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:28.826306 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:11:28.826351 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:28.826354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:28.826410 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:28.826502 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:28.827929 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.828753 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.830629 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.831041 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.831309 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.835180 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:28.835184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:28.835188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:28.835217 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:28.837791 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:28.838490 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:28.838497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:28.838501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:28.838625 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:28.839005 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:28.839011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:28.839023 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:28.839338 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:28.839498 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:28.839922 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 09:11:28.840251 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":713,"thread_start_us":573,"threads_started":1}
I20260806 09:11:28.840605 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:28.825392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39659 (local address 127.0.0.1:45702)
0806 09:11:28.825922 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:28.825947 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:28.826094 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:28.826577 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:28.826583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:28.827489 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:28.827763 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.827776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.828918 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.828924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.830446 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:28.830461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:28.831373 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:28.831380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:28.831562 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:28.832277 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:28.832302 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:28.834939 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:28.837935 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:28.837947 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:28.837956 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:28.838370 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:28.838734 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:28.838739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:28.838743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:28.838899 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:28.839466 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:28.839475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:28.839667 (+ 192us) client_negotiation.cc:769] Sending connection context
0806 09:11:28.839841 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 09:11:28.840129 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260806 09:11:28.845886 14080 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:45702:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:11:28.846971 14080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:28.854415 14265 tablet_service.cc:1467] Processing CreateTablet for tablet a83b8dab7cde482394acf04bd9619c09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6b7a9cfba1749a5b966096243550cc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:11:28.854437 14464 tablet_service.cc:1467] Processing CreateTablet for tablet a83b8dab7cde482394acf04bd9619c09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6b7a9cfba1749a5b966096243550cc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:11:28.854791 14667 tablet_service.cc:1467] Processing CreateTablet for tablet a83b8dab7cde482394acf04bd9619c09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f6b7a9cfba1749a5b966096243550cc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:11:28.855031 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83b8dab7cde482394acf04bd9619c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.855046 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83b8dab7cde482394acf04bd9619c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.855417 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83b8dab7cde482394acf04bd9619c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:28.862639 14802 tablet_bootstrap.cc:492] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Bootstrap starting.
I20260806 09:11:28.862676 14799 tablet_bootstrap.cc:492] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: Bootstrap starting.
I20260806 09:11:28.864257 14799 tablet_bootstrap.cc:654] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.864400 14802 tablet_bootstrap.cc:654] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.866657 14799 tablet_bootstrap.cc:492] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: No bootstrap required, opened a new log
I20260806 09:11:28.866798 14804 tablet_bootstrap.cc:492] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Bootstrap starting.
I20260806 09:11:28.866969 14799 ts_tablet_manager.cc:1397] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 09:11:28.867614 14799 raft_consensus.cc:348] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.867831 14799 raft_consensus.cc:374] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.867913 14799 raft_consensus.cc:729] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb87ebed0274353a9d4a27acc4fc0b7, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.868160 14799 consensus_queue.cc:260] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.868434 14804 tablet_bootstrap.cc:654] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:28.868713 14799 ts_tablet_manager.cc:1428] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 09:11:28.868607 14807 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:28.868979 14807 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:11:28.870496 14802 tablet_bootstrap.cc:492] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: No bootstrap required, opened a new log
I20260806 09:11:28.870666 14802 ts_tablet_manager.cc:1397] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260806 09:11:28.871789 14802 raft_consensus.cc:348] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.872051 14802 raft_consensus.cc:374] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.872179 14802 raft_consensus.cc:729] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.872470 14802 consensus_queue.cc:260] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.872911 14834 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:28.873135 14804 tablet_bootstrap.cc:492] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: No bootstrap required, opened a new log
I20260806 09:11:28.873226 14834 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:11:28.873294 14804 ts_tablet_manager.cc:1397] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260806 09:11:28.873708 14802 ts_tablet_manager.cc:1428] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:28.874128 14804 raft_consensus.cc:348] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.874377 14804 raft_consensus.cc:374] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:28.874472 14804 raft_consensus.cc:729] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a6bd8be7fa4c1482af1ded80bbcf9e, State: Initialized, Role: FOLLOWER
I20260806 09:11:28.875053 14804 consensus_queue.cc:260] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.875730 14804 ts_tablet_manager.cc:1428] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 09:11:28.875579 14808 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:28.875967 14808 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:11:28.967948 14834 raft_consensus.cc:482] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:28.968183 14834 raft_consensus.cc:504] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.968950 14834 leader_election.cc:288] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:28.969542 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83b8dab7cde482394acf04bd9619c09" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" is_pre_election: true
I20260806 09:11:28.969537 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83b8dab7cde482394acf04bd9619c09" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" is_pre_election: true
I20260806 09:11:28.969825 14704 raft_consensus.cc:2454] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 0.
I20260806 09:11:28.969828 14296 raft_consensus.cc:2454] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 0.
I20260806 09:11:28.970319 14399 leader_election.cc:302] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bca3e5ba3d4d7db69f7d69002b1b2e, afb87ebed0274353a9d4a27acc4fc0b7; no voters:
I20260806 09:11:28.970672 14834 raft_consensus.cc:2781] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:28.970809 14834 raft_consensus.cc:482] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:28.970956 14834 raft_consensus.cc:3037] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.973786 14834 raft_consensus.cc:504] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.974407 14834 leader_election.cc:288] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 election: Requested vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:28.974840 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83b8dab7cde482394acf04bd9619c09" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e"
I20260806 09:11:28.974938 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83b8dab7cde482394acf04bd9619c09" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb87ebed0274353a9d4a27acc4fc0b7"
I20260806 09:11:28.975145 14296 raft_consensus.cc:3037] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.975181 14704 raft_consensus.cc:3037] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:28.978112 14704 raft_consensus.cc:2454] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 1.
I20260806 09:11:28.978116 14296 raft_consensus.cc:2454] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 1.
I20260806 09:11:28.978597 14398 leader_election.cc:302] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02bca3e5ba3d4d7db69f7d69002b1b2e, afb87ebed0274353a9d4a27acc4fc0b7, d6a6bd8be7fa4c1482af1ded80bbcf9e; no voters:
I20260806 09:11:28.978976 14834 raft_consensus.cc:2781] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:28.979240 14834 raft_consensus.cc:686] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 LEADER]: Becoming Leader. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Running, Role: LEADER
I20260806 09:11:28.979501 14834 consensus_queue.cc:237] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:28.981923 14829 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: TxnStatusTablet state changed. Reason: New leader 02bca3e5ba3d4d7db69f7d69002b1b2e. Latest consensus state: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:28.982304 14829 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:11:28.983003 14863 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:11:28.982800 14082 catalog_manager.cc:5495] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2). New cstate: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } health_report { overall_health: UNKNOWN } } }
I20260806 09:11:28.983297 14863 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:11:29.010604 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.004428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:42335 (local address 127.0.0.1:53688)
0806 09:11:29.004696 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:29.004720 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:29.004910 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:29.005384 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:29.005390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:29.005404 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:11:29.005607 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.005619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.006905 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.006909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.008297 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.008312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.009252 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.009259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.009433 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.009503 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.009959 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.009986 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 09:11:29.010098 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:11:29.010271 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 09:11:29.010600 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.004441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.3.69.2:42335)
0806 09:11:29.004787 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.004792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.004813 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:11:29.005140 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.005149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.005229 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.005324 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:29.005790 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.006720 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.008469 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.009079 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.009236 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.009707 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.009839 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.009957 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.010128 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.010216 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260806 09:11:29.014645 14296 raft_consensus.cc:1261] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Refusing update from remote peer 02bca3e5ba3d4d7db69f7d69002b1b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:29.014750 14704 raft_consensus.cc:1261] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Refusing update from remote peer 02bca3e5ba3d4d7db69f7d69002b1b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:29.015424 14829 consensus_queue.cc:1035] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:29.015761 14834 consensus_queue.cc:1035] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:29.020960 14807 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: TxnStatusTablet state changed. Reason: New leader 02bca3e5ba3d4d7db69f7d69002b1b2e. Latest consensus state: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:29.021260 14807 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:11:29.022341 14829 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:29.022627 14829 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:11:29.023793 14822 mvcc.cc:205] Tried to move back new op lower bound from 7315486674997600256 to 7315486674862469120. Current Snapshot: MvccSnapshot[applied={T|T < 7315486674997600256}]
I20260806 09:11:29.023924 14819 mvcc.cc:205] Tried to move back new op lower bound from 7315486674997600256 to 7315486674862469120. Current Snapshot: MvccSnapshot[applied={T|T < 7315486674997600256 or (T in {7315486674997600256})}]
I20260806 09:11:29.024499 14807 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:29.024673 14807 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P afb87ebed0274353a9d4a27acc4fc0b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:11:29.027454 14808 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: TxnStatusTablet state changed. Reason: New leader 02bca3e5ba3d4d7db69f7d69002b1b2e. Latest consensus state: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:29.027743 14808 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:11:29.028951 14809 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:29.029246 14809 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P 02bca3e5ba3d4d7db69f7d69002b1b2e: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:11:29.030740 14808 tablet_replica.cc:439] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } } }
I20260806 09:11:29.030964 14808 tablet_replica.cc:442] T a83b8dab7cde482394acf04bd9619c09 P d6a6bd8be7fa4c1482af1ded80bbcf9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:11:29.048513 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.041820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:53693 (local address 127.3.69.2:42335)
0806 09:11:29.042065 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.042071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.042735 (+ 664us) server_negotiation.cc:407] Connection header received
0806 09:11:29.042886 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.042893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.042984 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.043177 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:29.043650 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.044813 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.046268 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.046980 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.047173 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.047663 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.047803 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.047909 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.048049 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.048143 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:11:29.048553 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.041839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:42335 (local address 127.3.69.2:53693)
0806 09:11:29.042618 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:29.042635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:29.042757 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:29.043164 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:29.043168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:29.043177 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:11:29.043501 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.043510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.044947 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.044950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.046122 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.046134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.047265 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.047270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.047403 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.047463 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.047924 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.047932 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 09:11:29.047989 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 09:11:29.048093 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":436,"threads_started":1}
I20260806 09:11:29.059283 14296 raft_consensus.cc:1261] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Refusing update from remote peer 02bca3e5ba3d4d7db69f7d69002b1b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:29.059297 14704 raft_consensus.cc:1261] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Refusing update from remote peer 02bca3e5ba3d4d7db69f7d69002b1b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:29.059873 14813 consensus_queue.cc:1035] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:29.060212 14834 consensus_queue.cc:1035] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:29.067245 14824 mvcc.cc:205] Tried to move back new op lower bound from 7315486675181391872 to 7315486674053668864. Current Snapshot: MvccSnapshot[applied={T|T < 7315486675181391872}]
I20260806 09:11:29.125339 14079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:29.125582 14079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:29.128643 14079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: Sending DeleteTablet for 3 replicas of tablet 1ca26e7d3ce14c8fb6eca9a414600f23
I20260806 09:11:29.129752 14265 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca26e7d3ce14c8fb6eca9a414600f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:56200
I20260806 09:11:29.129952 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca26e7d3ce14c8fb6eca9a414600f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37494
I20260806 09:11:29.130060 14838 tablet_replica.cc:330] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: stopping tablet replica
I20260806 09:11:29.130273 14838 raft_consensus.cc:2229] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:29.130278 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca26e7d3ce14c8fb6eca9a414600f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:53656
I20260806 09:11:29.130460 14838 raft_consensus.cc:2258] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.130561 14840 tablet_replica.cc:330] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: stopping tablet replica
I20260806 09:11:29.130745 14839 tablet_replica.cc:330] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: stopping tablet replica
I20260806 09:11:29.130748 14840 raft_consensus.cc:2229] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:11:29.130959 14839 raft_consensus.cc:2229] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:29.131160 14839 raft_consensus.cc:2258] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.131067 14840 raft_consensus.cc:2258] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.132479 14839 ts_tablet_manager.cc:1905] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.132534 14840 ts_tablet_manager.cc:1905] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.139386 14840 ts_tablet_manager.cc:1918] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:11:29.139637 14840 log.cc:1196] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/1ca26e7d3ce14c8fb6eca9a414600f23
I20260806 09:11:29.140108 14840 ts_tablet_manager.cc:1939] T 1ca26e7d3ce14c8fb6eca9a414600f23 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting consensus metadata
I20260806 09:11:29.140105 14839 ts_tablet_manager.cc:1918] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:11:29.140247 14839 log.cc:1196] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/1ca26e7d3ce14c8fb6eca9a414600f23
I20260806 09:11:29.140692 14839 ts_tablet_manager.cc:1939] T 1ca26e7d3ce14c8fb6eca9a414600f23 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting consensus metadata
I20260806 09:11:29.141788 14067 catalog_manager.cc:4841] TS 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2:42335): tablet 1ca26e7d3ce14c8fb6eca9a414600f23 (table test-table [id=e395f934ea2042eb9eecfd83df269e90]) successfully deleted
I20260806 09:11:29.142061 14067 catalog_manager.cc:4841] TS afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719): tablet 1ca26e7d3ce14c8fb6eca9a414600f23 (table test-table [id=e395f934ea2042eb9eecfd83df269e90]) successfully deleted
I20260806 09:11:29.142537 14838 ts_tablet_manager.cc:1905] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.151643 14838 ts_tablet_manager.cc:1918] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:11:29.151813 14838 log.cc:1196] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/1ca26e7d3ce14c8fb6eca9a414600f23
I20260806 09:11:29.152190 14838 ts_tablet_manager.cc:1939] T 1ca26e7d3ce14c8fb6eca9a414600f23 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting consensus metadata
I20260806 09:11:29.153183 14065 catalog_manager.cc:4841] TS d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655): tablet 1ca26e7d3ce14c8fb6eca9a414600f23 (table test-table [id=e395f934ea2042eb9eecfd83df269e90]) successfully deleted
Aug 06 09:11:29 dist-test-slave-w4v5 krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:11:29 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:29 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:11:29.210604 14857 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:29.197477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.3.69.62:39659)
0806 09:11:29.197727 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.197734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.197873 (+ 139us) server_negotiation.cc:407] Connection header received
0806 09:11:29.198083 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.198088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.198176 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.198290 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:29.200064 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.201254 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.203300 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.203975 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.204130 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.210324 (+ 6194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45714: BlockingRecv error: recv got EOF from 127.0.0.1:45714 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 09:11:29 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM
I20260806 09:11:29.232435 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.216435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.3.69.62:39659)
0806 09:11:29.216602 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.216608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.217064 (+ 456us) server_negotiation.cc:407] Connection header received
0806 09:11:29.217235 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.217240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.217322 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.217428 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:29.219016 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.220200 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.221452 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.222103 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.222255 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.226303 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:29.226311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:29.226316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:29.226361 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:29.229651 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:29.230353 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:29.230359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:29.230363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:29.230539 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:29.230908 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:29.230913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:29.230917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:29.231296 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:29.231556 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.231838 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.232072 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 09:11:29.236495 14079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:11:29.237218 14079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:29.244467 14265 tablet_service.cc:1467] Processing CreateTablet for tablet 5463286ea1ad4c2380126127838a0231 (DEFAULT_TABLE table=test-table [id=069e0ea541284092a2620b6f16e52b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:29.244977 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5463286ea1ad4c2380126127838a0231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:29.244972 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 5463286ea1ad4c2380126127838a0231 (DEFAULT_TABLE table=test-table [id=069e0ea541284092a2620b6f16e52b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:29.245210 14667 tablet_service.cc:1467] Processing CreateTablet for tablet 5463286ea1ad4c2380126127838a0231 (DEFAULT_TABLE table=test-table [id=069e0ea541284092a2620b6f16e52b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:29.245400 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5463286ea1ad4c2380126127838a0231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:29.245639 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5463286ea1ad4c2380126127838a0231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:29.251976 14804 tablet_bootstrap.cc:492] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Bootstrap starting.
I20260806 09:11:29.252206 14802 tablet_bootstrap.cc:492] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Bootstrap starting.
I20260806 09:11:29.253358 14804 tablet_bootstrap.cc:654] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:29.253890 14802 tablet_bootstrap.cc:654] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:29.256131 14804 tablet_bootstrap.cc:492] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: No bootstrap required, opened a new log
I20260806 09:11:29.256286 14804 ts_tablet_manager.cc:1397] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 09:11:29.256914 14799 tablet_bootstrap.cc:492] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: Bootstrap starting.
I20260806 09:11:29.257560 14804 raft_consensus.cc:348] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.257844 14804 raft_consensus.cc:374] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:29.257933 14804 raft_consensus.cc:729] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a6bd8be7fa4c1482af1ded80bbcf9e, State: Initialized, Role: FOLLOWER
I20260806 09:11:29.258199 14804 consensus_queue.cc:260] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.258631 14799 tablet_bootstrap.cc:654] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:29.262202 14804 ts_tablet_manager.cc:1428] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260806 09:11:29.265636 14802 tablet_bootstrap.cc:492] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: No bootstrap required, opened a new log
I20260806 09:11:29.265874 14802 ts_tablet_manager.cc:1397] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260806 09:11:29.266832 14802 raft_consensus.cc:348] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.267141 14802 raft_consensus.cc:374] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:29.267299 14802 raft_consensus.cc:729] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Initialized, Role: FOLLOWER
I20260806 09:11:29.267588 14802 consensus_queue.cc:260] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.268292 14802 ts_tablet_manager.cc:1428] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 09:11:29.270041 14799 tablet_bootstrap.cc:492] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: No bootstrap required, opened a new log
I20260806 09:11:29.270263 14799 ts_tablet_manager.cc:1397] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260806 09:11:29.271149 14799 raft_consensus.cc:348] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.271431 14799 raft_consensus.cc:374] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:29.271569 14799 raft_consensus.cc:729] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb87ebed0274353a9d4a27acc4fc0b7, State: Initialized, Role: FOLLOWER
I20260806 09:11:29.271844 14799 consensus_queue.cc:260] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.272464 14799 ts_tablet_manager.cc:1428] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 09:11:29.533591 14813 raft_consensus.cc:482] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:29.533806 14813 raft_consensus.cc:504] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.534583 14813 leader_election.cc:288] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:29.535271 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5463286ea1ad4c2380126127838a0231" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" is_pre_election: true
I20260806 09:11:29.535333 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5463286ea1ad4c2380126127838a0231" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" is_pre_election: true
I20260806 09:11:29.535555 14296 raft_consensus.cc:2454] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 0.
I20260806 09:11:29.535624 14704 raft_consensus.cc:2454] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 0.
I20260806 09:11:29.536046 14398 leader_election.cc:302] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bca3e5ba3d4d7db69f7d69002b1b2e, d6a6bd8be7fa4c1482af1ded80bbcf9e; no voters:
I20260806 09:11:29.536351 14813 raft_consensus.cc:2781] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:29.536516 14813 raft_consensus.cc:482] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:29.536643 14813 raft_consensus.cc:3037] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:29.538946 14813 raft_consensus.cc:504] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.539698 14813 leader_election.cc:288] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 election: Requested vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719)
I20260806 09:11:29.540203 14704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5463286ea1ad4c2380126127838a0231" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb87ebed0274353a9d4a27acc4fc0b7"
I20260806 09:11:29.540208 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5463286ea1ad4c2380126127838a0231" candidate_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e"
I20260806 09:11:29.540414 14704 raft_consensus.cc:3037] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:29.540414 14296 raft_consensus.cc:3037] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:29.542623 14704 raft_consensus.cc:2454] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 1.
I20260806 09:11:29.543043 14399 leader_election.cc:302] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bca3e5ba3d4d7db69f7d69002b1b2e, afb87ebed0274353a9d4a27acc4fc0b7; no voters:
I20260806 09:11:29.543388 14813 raft_consensus.cc:2781] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:29.543489 14296 raft_consensus.cc:2454] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bca3e5ba3d4d7db69f7d69002b1b2e in term 1.
I20260806 09:11:29.543565 14813 raft_consensus.cc:686] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 LEADER]: Becoming Leader. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Running, Role: LEADER
I20260806 09:11:29.543912 14813 consensus_queue.cc:237] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.546958 14082 catalog_manager.cc:5495] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2). New cstate: current_term: 1 leader_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } health_report { overall_health: UNKNOWN } } }
I20260806 09:11:29.564282 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.557951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.3.69.2:42335)
0806 09:11:29.558515 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.558532 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.558559 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:11:29.558661 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.558665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.558741 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.558825 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:29.559302 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.560382 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.562402 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.562862 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.562959 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.563516 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.563675 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.563748 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.563886 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.563946 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":306,"threads_started":1}
I20260806 09:11:29.566905 14296 raft_consensus.cc:1261] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Refusing update from remote peer 02bca3e5ba3d4d7db69f7d69002b1b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:29.567018 14704 raft_consensus.cc:1261] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Refusing update from remote peer 02bca3e5ba3d4d7db69f7d69002b1b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:29.567713 14829 consensus_queue.cc:1035] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:29.567986 14813 consensus_queue.cc:1035] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:29.580549 14079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:29.580817 14079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:29.585500 14079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: Sending DeleteTablet for 3 replicas of tablet 5463286ea1ad4c2380126127838a0231
I20260806 09:11:29.586336 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet 5463286ea1ad4c2380126127838a0231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37494
I20260806 09:11:29.586378 14265 tablet_service.cc:1514] Processing DeleteTablet for tablet 5463286ea1ad4c2380126127838a0231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:56200
I20260806 09:11:29.586675 14839 tablet_replica.cc:330] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: stopping tablet replica
I20260806 09:11:29.586675 14838 tablet_replica.cc:330] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: stopping tablet replica
I20260806 09:11:29.586885 14839 raft_consensus.cc:2229] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:29.586953 14838 raft_consensus.cc:2229] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:29.587191 14839 raft_consensus.cc:2258] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.587251 14838 raft_consensus.cc:2258] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.588366 14839 ts_tablet_manager.cc:1905] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.588423 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 5463286ea1ad4c2380126127838a0231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:53656
I20260806 09:11:29.588747 14840 tablet_replica.cc:330] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: stopping tablet replica
I20260806 09:11:29.588939 14840 raft_consensus.cc:2229] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:11:29.588977 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet 5463286ea1ad4c2380126127838a0231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37494
I20260806 09:11:29.589305 14840 raft_consensus.cc:2258] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.590739 14079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:11:29.591456 14079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:11:29.591951 14840 ts_tablet_manager.cc:1905] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.600533 14265 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1f28f7e5de479c8f9cdf748fae5d36 (DEFAULT_TABLE table=test-table [id=6f5092c53cc04af79bb8ba6702631dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:29.600824 14466 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1f28f7e5de479c8f9cdf748fae5d36 (DEFAULT_TABLE table=test-table [id=6f5092c53cc04af79bb8ba6702631dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:29.601017 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1f28f7e5de479c8f9cdf748fae5d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:29.601256 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1f28f7e5de479c8f9cdf748fae5d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:29.604609 14839 ts_tablet_manager.cc:1918] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:29.604764 14839 log.cc:1196] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/5463286ea1ad4c2380126127838a0231
I20260806 09:11:29.607025 14840 ts_tablet_manager.cc:1918] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:29.607191 14840 log.cc:1196] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/5463286ea1ad4c2380126127838a0231
I20260806 09:11:29.608150 14838 ts_tablet_manager.cc:1905] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.608192 14667 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1f28f7e5de479c8f9cdf748fae5d36 (DEFAULT_TABLE table=test-table [id=6f5092c53cc04af79bb8ba6702631dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:11:29.608278 14804 tablet_bootstrap.cc:492] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Bootstrap starting.
I20260806 09:11:29.608567 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1f28f7e5de479c8f9cdf748fae5d36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:29.610080 14804 tablet_bootstrap.cc:654] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:29.615693 14799 tablet_bootstrap.cc:492] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: Bootstrap starting.
I20260806 09:11:29.616223 14838 ts_tablet_manager.cc:1918] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:11:29.616377 14838 log.cc:1196] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/5463286ea1ad4c2380126127838a0231
I20260806 09:11:29.616914 14838 ts_tablet_manager.cc:1939] T 5463286ea1ad4c2380126127838a0231 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting consensus metadata
I20260806 09:11:29.617244 14799 tablet_bootstrap.cc:654] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:29.618114 14065 catalog_manager.cc:4841] TS d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655): tablet 5463286ea1ad4c2380126127838a0231 (table test-table [id=069e0ea541284092a2620b6f16e52b74]) successfully deleted
I20260806 09:11:29.618314 14839 ts_tablet_manager.cc:1939] T 5463286ea1ad4c2380126127838a0231 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting consensus metadata
I20260806 09:11:29.619139 14804 tablet_bootstrap.cc:492] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: No bootstrap required, opened a new log
I20260806 09:11:29.619374 14804 ts_tablet_manager.cc:1397] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260806 09:11:29.619505 14067 catalog_manager.cc:4841] TS afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719): tablet 5463286ea1ad4c2380126127838a0231 (table test-table [id=069e0ea541284092a2620b6f16e52b74]) successfully deleted
I20260806 09:11:29.620370 14840 ts_tablet_manager.cc:1939] T 5463286ea1ad4c2380126127838a0231 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting consensus metadata
I20260806 09:11:29.620362 14804 raft_consensus.cc:348] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.620652 14804 raft_consensus.cc:374] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:29.620733 14804 raft_consensus.cc:729] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a6bd8be7fa4c1482af1ded80bbcf9e, State: Initialized, Role: FOLLOWER
I20260806 09:11:29.620764 14802 tablet_bootstrap.cc:492] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Bootstrap starting.
I20260806 09:11:29.620987 14804 consensus_queue.cc:260] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.621515 14067 catalog_manager.cc:4841] TS 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2:42335): tablet 5463286ea1ad4c2380126127838a0231 (table test-table [id=069e0ea541284092a2620b6f16e52b74]) successfully deleted
W20260806 09:11:29.622134 14067 catalog_manager.cc:4808] TS afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719): delete failed for tablet 5463286ea1ad4c2380126127838a0231 because the tablet was not found. No further retry: Not found: Tablet not found: 5463286ea1ad4c2380126127838a0231
I20260806 09:11:29.622293 14804 ts_tablet_manager.cc:1428] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 09:11:29.622486 14802 tablet_bootstrap.cc:654] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:29.624467 14802 tablet_bootstrap.cc:492] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: No bootstrap required, opened a new log
I20260806 09:11:29.624682 14802 ts_tablet_manager.cc:1397] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260806 09:11:29.624912 14799 tablet_bootstrap.cc:492] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: No bootstrap required, opened a new log
I20260806 09:11:29.625082 14799 ts_tablet_manager.cc:1397] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20260806 09:11:29.625402 14802 raft_consensus.cc:348] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.625624 14802 raft_consensus.cc:374] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:29.625708 14802 raft_consensus.cc:729] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bca3e5ba3d4d7db69f7d69002b1b2e, State: Initialized, Role: FOLLOWER
I20260806 09:11:29.625876 14799 raft_consensus.cc:348] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.626076 14799 raft_consensus.cc:374] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:29.625950 14802 consensus_queue.cc:260] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.626133 14799 raft_consensus.cc:729] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb87ebed0274353a9d4a27acc4fc0b7, State: Initialized, Role: FOLLOWER
I20260806 09:11:29.626423 14802 ts_tablet_manager.cc:1428] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 09:11:29.627146 14799 consensus_queue.cc:260] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.627938 14799 ts_tablet_manager.cc:1428] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 09:11:29.670341 14807 raft_consensus.cc:482] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:11:29.670552 14807 raft_consensus.cc:504] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.672303 14807 leader_election.cc:288] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2:42335)
I20260806 09:11:29.679051 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.673103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:56479 (local address 127.3.69.2:42335)
0806 09:11:29.673300 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.673306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.673975 (+ 669us) server_negotiation.cc:407] Connection header received
0806 09:11:29.674163 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.674168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.674237 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.674335 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:29.674868 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.675646 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.677374 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.678151 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.678295 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.678406 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.678681 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.678802 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:11:29.679430 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.672671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:46119 (local address 127.3.69.1:42655)
0806 09:11:29.674199 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.674204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.674221 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:11:29.674277 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.674281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.674333 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.674454 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:11:29.675124 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.675865 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.677591 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.678389 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.678537 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.678648 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.678944 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.679092 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":1270,"threads_started":1}
I20260806 09:11:29.679571 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.672713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:42335 (local address 127.3.69.3:56479)
0806 09:11:29.673876 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:29.673889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:29.674027 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:29.674356 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:29.674361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:29.674386 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:29.674706 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.674715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.675793 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.675799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.677228 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.677240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.678350 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.678356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.678517 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.678544 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 09:11:29.678655 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 09:11:29.678786 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":1,"spinlock_wait_cycles":32000,"thread_start_us":603,"threads_started":1}
I20260806 09:11:29.679618 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.672399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42655 (local address 127.3.69.3:46119)
0806 09:11:29.673413 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:29.673433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:29.673597 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:29.674477 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:29.674482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:29.674541 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:11:29.674929 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.674937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.676000 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.676008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.677417 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.677428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.678607 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.678613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.678773 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.678796 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 09:11:29.678909 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 09:11:29.679034 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":1,"spinlock_wait_cycles":6144,"thread_start_us":1212,"threads_started":1}
I20260806 09:11:29.680289 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1f28f7e5de479c8f9cdf748fae5d36" candidate_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" is_pre_election: true
I20260806 09:11:29.680356 14503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1f28f7e5de479c8f9cdf748fae5d36" candidate_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" is_pre_election: true
I20260806 09:11:29.680527 14296 raft_consensus.cc:2454] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb87ebed0274353a9d4a27acc4fc0b7 in term 0.
I20260806 09:11:29.680745 14503 raft_consensus.cc:2454] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb87ebed0274353a9d4a27acc4fc0b7 in term 0.
I20260806 09:11:29.681066 14601 leader_election.cc:302] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb87ebed0274353a9d4a27acc4fc0b7, d6a6bd8be7fa4c1482af1ded80bbcf9e; no voters:
I20260806 09:11:29.681473 14807 raft_consensus.cc:2781] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:11:29.681607 14807 raft_consensus.cc:482] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:11:29.681726 14807 raft_consensus.cc:3037] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:29.683966 14807 raft_consensus.cc:504] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.684597 14807 leader_election.cc:288] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [CANDIDATE]: Term 1 election: Requested vote from peers d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655), 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2:42335)
I20260806 09:11:29.685053 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1f28f7e5de479c8f9cdf748fae5d36" candidate_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e"
I20260806 09:11:29.685160 14503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1f28f7e5de479c8f9cdf748fae5d36" candidate_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e"
I20260806 09:11:29.685278 14296 raft_consensus.cc:3037] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:29.685379 14503 raft_consensus.cc:3037] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:29.687595 14296 raft_consensus.cc:2454] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb87ebed0274353a9d4a27acc4fc0b7 in term 1.
I20260806 09:11:29.687832 14503 raft_consensus.cc:2454] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb87ebed0274353a9d4a27acc4fc0b7 in term 1.
I20260806 09:11:29.688030 14601 leader_election.cc:302] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb87ebed0274353a9d4a27acc4fc0b7, d6a6bd8be7fa4c1482af1ded80bbcf9e; no voters:
I20260806 09:11:29.688359 14807 raft_consensus.cc:2781] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:29.688796 14807 raft_consensus.cc:686] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 LEADER]: Becoming Leader. State: Replica: afb87ebed0274353a9d4a27acc4fc0b7, State: Running, Role: LEADER
I20260806 09:11:29.689265 14807 consensus_queue.cc:237] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } }
I20260806 09:11:29.692719 14079 catalog_manager.cc:5495] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3). New cstate: current_term: 1 leader_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afb87ebed0274353a9d4a27acc4fc0b7" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41719 } health_report { overall_health: HEALTHY } } }
I20260806 09:11:29.718818 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.713222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.3.69.3:41719)
0806 09:11:29.713819 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.713824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.713843 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:11:29.713923 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.713926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.714005 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.714150 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:29.714509 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.715428 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.716691 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.717133 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.717231 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.717711 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.718284 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.718409 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.718486 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.718548 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":370,"threads_started":1}
I20260806 09:11:29.733212 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.728126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:49273 (local address 127.3.69.2:42335)
0806 09:11:29.728280 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.728284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.728878 (+ 594us) server_negotiation.cc:407] Connection header received
0806 09:11:29.729018 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.729022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.729087 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.729172 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:29.729606 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.730313 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.731679 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.732079 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.732193 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.732567 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.732685 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.732753 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.732924 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.732989 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 09:11:29.733300 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.728125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:42335 (local address 127.3.69.3:49273)
0806 09:11:29.728779 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:29.728794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:29.728912 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:29.729200 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:29.729203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:29.729212 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:11:29.729469 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.729475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.730434 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.730437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.731555 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.731567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.732249 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.732253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.732367 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.732417 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.732788 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.732801 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 09:11:29.732893 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 09:11:29.733012 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":323,"threads_started":1}
I20260806 09:11:29.741930 14296 raft_consensus.cc:1261] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Refusing update from remote peer afb87ebed0274353a9d4a27acc4fc0b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:29.742105 14503 raft_consensus.cc:1261] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Refusing update from remote peer afb87ebed0274353a9d4a27acc4fc0b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:11:29.742645 14920 consensus_queue.cc:1035] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a6bd8be7fa4c1482af1ded80bbcf9e" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:29.743113 14807 consensus_queue.cc:1035] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02bca3e5ba3d4d7db69f7d69002b1b2e" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 42335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:11:29.771893 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.766655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:54671 (local address 127.3.69.3:41719)
0806 09:11:29.766945 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.766949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.767140 (+ 191us) server_negotiation.cc:407] Connection header received
0806 09:11:29.767263 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.767266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.767326 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.767394 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:11:29.767724 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.768505 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.770115 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.770606 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.770741 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.771154 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.771248 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.771348 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.771495 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.771578 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260806 09:11:29.772119 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.766428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41719 (local address 127.3.69.2:54671)
0806 09:11:29.767036 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:29.767053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:29.767177 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:29.767410 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:29.767412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:29.767421 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:11:29.767601 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.767608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.768668 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.768676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.769973 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:29.769986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.770751 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.770757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:29.770913 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.770948 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:11:29.771349 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:11:29.771364 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 09:11:29.771462 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 09:11:29.771596 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":311,"threads_started":1}
I20260806 09:11:29.803934 14082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:29.804234 14082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:11:29.807585 14082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d649e5b6c584cf9a8060d2cece13db2: Sending DeleteTablet for 3 replicas of tablet 1a1f28f7e5de479c8f9cdf748fae5d36
I20260806 09:11:29.808353 14265 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1f28f7e5de479c8f9cdf748fae5d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:56200
I20260806 09:11:29.808677 14838 tablet_replica.cc:330] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: stopping tablet replica
I20260806 09:11:29.808761 14466 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1f28f7e5de479c8f9cdf748fae5d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:53656
I20260806 09:11:29.808882 14838 raft_consensus.cc:2229] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:29.809067 14838 raft_consensus.cc:2258] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.809070 14840 tablet_replica.cc:330] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: stopping tablet replica
I20260806 09:11:29.809271 14840 raft_consensus.cc:2229] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:11:29.809451 14840 raft_consensus.cc:2258] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.809775 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1f28f7e5de479c8f9cdf748fae5d36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:11:29 UTC) from {username='oryx', principal='oryx/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37494
I20260806 09:11:29.810084 14839 tablet_replica.cc:330] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: stopping tablet replica
I20260806 09:11:29.810324 14839 raft_consensus.cc:2229] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:11:29.810794 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 14176
I20260806 09:11:29.810870 14839 raft_consensus.cc:2258] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:11:29.810941 14838 ts_tablet_manager.cc:1905] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.812106 14839 ts_tablet_manager.cc:1905] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.815416 14840 ts_tablet_manager.cc:1905] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:11:29.821199 14839 ts_tablet_manager.cc:1918] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:11:29.821389 14839 log.cc:1196] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/1a1f28f7e5de479c8f9cdf748fae5d36
I20260806 09:11:29.821986 14839 ts_tablet_manager.cc:1939] T 1a1f28f7e5de479c8f9cdf748fae5d36 P afb87ebed0274353a9d4a27acc4fc0b7: Deleting consensus metadata
I20260806 09:11:29.822033 14840 ts_tablet_manager.cc:1918] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:11:29.822198 14840 log.cc:1196] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/1a1f28f7e5de479c8f9cdf748fae5d36
I20260806 09:11:29.822722 14840 ts_tablet_manager.cc:1939] T 1a1f28f7e5de479c8f9cdf748fae5d36 P 02bca3e5ba3d4d7db69f7d69002b1b2e: Deleting consensus metadata
I20260806 09:11:29.823518 14067 catalog_manager.cc:4841] TS afb87ebed0274353a9d4a27acc4fc0b7 (127.3.69.3:41719): tablet 1a1f28f7e5de479c8f9cdf748fae5d36 (table test-table [id=6f5092c53cc04af79bb8ba6702631dc4]) successfully deleted
I20260806 09:11:29.828014 14838 ts_tablet_manager.cc:1918] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:11:29.828064 14067 catalog_manager.cc:4841] TS 02bca3e5ba3d4d7db69f7d69002b1b2e (127.3.69.2:42335): tablet 1a1f28f7e5de479c8f9cdf748fae5d36 (table test-table [id=6f5092c53cc04af79bb8ba6702631dc4]) successfully deleted
I20260806 09:11:29.828197 14838 log.cc:1196] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/1a1f28f7e5de479c8f9cdf748fae5d36
I20260806 09:11:29.828672 14838 ts_tablet_manager.cc:1939] T 1a1f28f7e5de479c8f9cdf748fae5d36 P d6a6bd8be7fa4c1482af1ded80bbcf9e: Deleting consensus metadata
I20260806 09:11:29.829856 14065 catalog_manager.cc:4841] TS d6a6bd8be7fa4c1482af1ded80bbcf9e (127.3.69.1:42655): tablet 1a1f28f7e5de479c8f9cdf748fae5d36 (table test-table [id=6f5092c53cc04af79bb8ba6702631dc4]) successfully deleted
Aug 06 09:11:29 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.1@KRBTEST.COM
I20260806 09:11:29.841739 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.813895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.3.69.1:42655)
0806 09:11:29.814085 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.814091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.826223 (+ 12132us) server_negotiation.cc:407] Connection header received
0806 09:11:29.826375 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.826381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.826469 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.826579 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:29.828133 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.829146 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.830677 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.831192 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.831338 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.835623 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:29.835636 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:29.835652 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:29.835683 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:29.838786 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:29.839607 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:29.839615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:29.839617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:29.839749 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:29.840165 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:29.840169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:29.840171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:29.840535 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:29.840850 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.841180 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.841432 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:11:29.842437 14359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:29.941628 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 14176
I20260806 09:11:29.967392 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 14374
Aug 06 09:11:29 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.2@KRBTEST.COM
I20260806 09:11:29.983237 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:29.968326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.3.69.2:42335)
0806 09:11:29.968502 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:29.968508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:29.968530 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:11:29.968630 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:29.968634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:29.968697 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:29.968824 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:29.970099 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.970794 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.972117 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:29.972553 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:29.972699 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:29.977203 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:29.977217 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:29.977233 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:29.977271 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:29.980334 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:29.981092 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:29.981100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:29.981103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:29.981245 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:29.981635 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:29.981641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:29.981644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:29.982002 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:29.982339 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:29.982651 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 09:11:29.982889 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 09:11:29.983939 14564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:30.135215 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 14374
I20260806 09:11:30.179304 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 14574
Aug 06 09:11:30 dist-test-slave-w4v5 krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.69.3@KRBTEST.COM
I20260806 09:11:30.201018 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:30.180244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.3.69.3:41719)
0806 09:11:30.180480 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:30.180488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:30.180651 (+ 163us) server_negotiation.cc:407] Connection header received
0806 09:11:30.180882 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:30.180888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:30.181002 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:30.181149 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:30.182868 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:30.184123 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:30.186153 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:30.186894 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:30.187026 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:30.193623 (+ 6597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:30.193641 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:30.193669 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:30.193721 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:30.197867 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:30.198668 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:30.198676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:30.198680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:30.198843 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:30.199283 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:30.199288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:30.199291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:30.199724 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:30.200026 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:30.200397 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 09:11:30.200606 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:11:30.201704 14762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:30.341686 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 14574
I20260806 09:11:30.374861 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 14041
I20260806 09:11:30.376197 14133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:30.492688 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 14041
2026-08-06T09:11:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6067 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:11:30 dist-test-slave-w4v5 krb5kdc[14949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:11:30 dist-test-slave-w4v5 krb5kdc[14949](info): set up 2 sockets
Aug 06 09:11:30 dist-test-slave-w4v5 krb5kdc[14949](info): commencing operation
krb5kdc: starting...
W20260806 09:11:31.816186 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:11:31 dist-test-slave-w4v5 krb5kdc[14949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:11:32.080807 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:38355
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:38355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:32.414793 14964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:32.415141 14964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:32.415323 14964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:32.422502 14964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:32.422645 14964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:32.422758 14964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:32.422863 14964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:32.422956 14964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:32.432726 14964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:38355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:38355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:32.434053 14964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:32.435413 14964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:32.447152 14969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:32.447223 14970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:32.447376 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:32.448285 14964 server_base.cc:1034] running on GCE node
I20260806 09:11:32.449057 14964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:32.450404 14964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:32.451679 14964 hybrid_clock.cc:648] HybridClock initialized: now 1786007492451639 us; error 48 us; skew 500 ppm
I20260806 09:11:32.452019 14964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T09:11:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2034 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:11:32 dist-test-slave-w4v5 krb5kdc[14981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:11:32 dist-test-slave-w4v5 krb5kdc[14981](info): set up 2 sockets
Aug 06 09:11:32 dist-test-slave-w4v5 krb5kdc[14981](info): commencing operation
krb5kdc: starting...
W20260806 09:11:33.889691 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:11:34 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:11:34.146906 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39371
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:34.480917 14997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:34.481294 14997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:34.481441 14997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:34.488482 14997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:34.488585 14997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:34.488677 14997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:34.488765 14997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:34.488830 14997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:34.498236 14997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:39371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:34.499596 14997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:34.500840 14997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:34.516980 15010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:34.517652 15008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:34.518359 14997 server_base.cc:1034] running on GCE node
W20260806 09:11:34.518577 15014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:34.519702 14997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:34.521553 14997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:34.522821 14997 hybrid_clock.cc:648] HybridClock initialized: now 1786007494522758 us; error 51 us; skew 500 ppm
I20260806 09:11:34.523348 14997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:34 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:34.528213 14997 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:11:34.530272 14997 webserver.cc:460] Webserver started at http://127.3.69.62:37277/ using document root <none> and password file <none>
I20260806 09:11:34.531303 14997 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:34.531491 14997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:34.531903 14997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:34.536015 14997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "db7e165a38e744ca94124452b9ef9fc6"
format_stamp: "Formatted at 2026-08-06 09:11:34 on dist-test-slave-w4v5"
server_key: "38d4d82f15d9636c8746b70be3c5d091"
server_key_iv: "62cee980d2b5cf2f2862a3db16c61759"
server_key_version: "encryptionkey@0"
I20260806 09:11:34.537211 14997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "db7e165a38e744ca94124452b9ef9fc6"
format_stamp: "Formatted at 2026-08-06 09:11:34 on dist-test-slave-w4v5"
server_key: "38d4d82f15d9636c8746b70be3c5d091"
server_key_iv: "62cee980d2b5cf2f2862a3db16c61759"
server_key_version: "encryptionkey@0"
I20260806 09:11:34.544776 14997 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260806 09:11:34.550154 15022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:34.551584 14997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:34.551801 14997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "db7e165a38e744ca94124452b9ef9fc6"
format_stamp: "Formatted at 2026-08-06 09:11:34 on dist-test-slave-w4v5"
server_key: "38d4d82f15d9636c8746b70be3c5d091"
server_key_iv: "62cee980d2b5cf2f2862a3db16c61759"
server_key_version: "encryptionkey@0"
I20260806 09:11:34.552017 14997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:34.574234 14997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:34.578539 14997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:34.578845 14997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:34.599574 14997 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:39371
I20260806 09:11:34.599615 15091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:39371 every 8 connection(s)
I20260806 09:11:34.601027 14997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:34.604828 15092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:34.611282 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 14997
I20260806 09:11:34.611508 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:34.612413 3348 external_mini_cluster.cc:1428] Setting key 12fef2053ff34946ad6c9d21c9effabb
I20260806 09:11:34.616880 15092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Bootstrap starting.
I20260806 09:11:34.620033 15092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:34.621008 15092 log.cc:824] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:34.623754 15092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: No bootstrap required, opened a new log
Aug 06 09:11:34 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:34.631642 15092 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } }
I20260806 09:11:34.632042 15092 raft_consensus.cc:374] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:34.632169 15092 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7e165a38e744ca94124452b9ef9fc6, State: Initialized, Role: FOLLOWER
I20260806 09:11:34.632822 15092 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } }
I20260806 09:11:34.633072 15092 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:34.633219 15092 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:34.633417 15092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:34.636039 15092 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } }
I20260806 09:11:34.636165 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:34.614613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.3.69.62:39371)
0806 09:11:34.616967 (+ 2354us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:34.616979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:34.617044 (+ 65us) server_negotiation.cc:407] Connection header received
0806 09:11:34.618787 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:34.618809 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:34.619236 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:34.619596 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:34.621116 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:34.622489 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:34.623832 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:34.624569 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:34.624775 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:34.628717 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:34.628728 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:34.628742 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:34.628784 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:34.631909 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:34.632696 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:34.632705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:34.632711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:34.632874 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:34.633276 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:34.633280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:34.633282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:34.633903 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:34.634268 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:34.634719 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 09:11:34.635204 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2106,"thread_start_us":1863,"threads_started":1}
I20260806 09:11:34.636704 15092 leader_election.cc:302] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7e165a38e744ca94124452b9ef9fc6; no voters:
I20260806 09:11:34.637328 15092 leader_election.cc:288] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:34.637449 15099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:34.639923 15099 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 1 LEADER]: Becoming Leader. State: Replica: db7e165a38e744ca94124452b9ef9fc6, State: Running, Role: LEADER
I20260806 09:11:34.640103 15092 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:34.640432 15099 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } }
I20260806 09:11:34.643033 15100 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db7e165a38e744ca94124452b9ef9fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } } }
I20260806 09:11:34.643432 15100 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:34.643903 15101 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7e165a38e744ca94124452b9ef9fc6. Latest consensus state: current_term: 1 leader_uuid: "db7e165a38e744ca94124452b9ef9fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } } }
I20260806 09:11:34.645112 15101 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:34.646296 15109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:34.652735 15109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:34.663736 15109 catalog_manager.cc:1347] Generated new cluster ID: ed96e926a6be4f3695b2259dec13d8be
I20260806 09:11:34.663851 15109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:34.683267 15109 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:34.684522 15109 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:34.700451 15109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Generated new TSK 0
I20260806 09:11:34.701318 15109 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:11:34.763867 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39371
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:35.127993 15126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:35.128260 15126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:35.128419 15126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:35.135612 15126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:35.135761 15126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:35.135949 15126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:35.145820 15126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:35.147035 15126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:35.148379 15126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:35.161453 15133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:35.161671 15132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:35.162092 15135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:35.162303 15126 server_base.cc:1034] running on GCE node
I20260806 09:11:35.162914 15126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:35.163856 15126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:35.165098 15126 hybrid_clock.cc:648] HybridClock initialized: now 1786007495165057 us; error 49 us; skew 500 ppm
I20260806 09:11:35.165406 15126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:35 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:35.168589 15126 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:11:35.169957 15126 webserver.cc:460] Webserver started at http://127.3.69.1:45493/ using document root <none> and password file <none>
I20260806 09:11:35.170640 15126 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:35.170787 15126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:35.171178 15126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:35.174039 15126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "64387a9987334a36a5a1d04ebdde95fb"
format_stamp: "Formatted at 2026-08-06 09:11:35 on dist-test-slave-w4v5"
server_key: "6c53a45f9965cdcffe266bfe32f61940"
server_key_iv: "48d3ca04410e33676bd765cc72b2b457"
server_key_version: "encryptionkey@0"
I20260806 09:11:35.174870 15126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "64387a9987334a36a5a1d04ebdde95fb"
format_stamp: "Formatted at 2026-08-06 09:11:35 on dist-test-slave-w4v5"
server_key: "6c53a45f9965cdcffe266bfe32f61940"
server_key_iv: "48d3ca04410e33676bd765cc72b2b457"
server_key_version: "encryptionkey@0"
I20260806 09:11:35.179826 15126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 09:11:35.183375 15142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:35.184599 15126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:11:35.184777 15126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "64387a9987334a36a5a1d04ebdde95fb"
format_stamp: "Formatted at 2026-08-06 09:11:35 on dist-test-slave-w4v5"
server_key: "6c53a45f9965cdcffe266bfe32f61940"
server_key_iv: "48d3ca04410e33676bd765cc72b2b457"
server_key_version: "encryptionkey@0"
I20260806 09:11:35.184969 15126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:35.198748 15126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:35.203171 15126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:35.203495 15126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:35.204423 15126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:35.205708 15126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:35.205808 15126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:35.205955 15126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:35.206032 15126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:35.239743 15126 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:39791
I20260806 09:11:35.239750 15307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:39791 every 8 connection(s)
I20260806 09:11:35.241241 15126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:35.244490 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 15126
I20260806 09:11:35.244710 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:11:35.245596 3348 external_mini_cluster.cc:1428] Setting key 46798e75b34fe7e5d40c41d418dc336a
Aug 06 09:11:35 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:35.261976 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:35.243943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:37283 (local address 127.3.69.62:39371)
0806 09:11:35.244482 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:35.244486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:35.245495 (+ 1009us) server_negotiation.cc:407] Connection header received
0806 09:11:35.247643 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:35.247648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:35.247747 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:35.247864 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:35.249376 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:35.250108 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:35.251734 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:35.252135 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:35.252276 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:35.256093 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:35.256097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:35.256101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:35.256133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:35.258881 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:35.259657 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:35.259663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:35.259665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:35.259794 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:35.260230 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:35.260241 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:35.260244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:35.260498 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:35.260671 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:35.261374 (+ 703us) server_negotiation.cc:299] Negotiation successful
0806 09:11:35.261601 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":305,"threads_started":1}
I20260806 09:11:35.262933 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:35.244520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:37283)
0806 09:11:35.245331 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:35.245375 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:35.247375 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:35.248109 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:35.248117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:35.248625 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:35.249208 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:35.249225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:35.250264 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:35.250269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:35.251588 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:35.251600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:35.252377 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:35.252387 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:35.252612 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:35.253319 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:35.253341 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:35.255886 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:35.259036 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:35.259045 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:35.259061 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:35.259523 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:35.259908 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:35.259914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:35.259918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:35.260106 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:35.260630 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:35.260640 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:35.260995 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 09:11:35.261263 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 09:11:35.261781 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":342,"threads_started":1}
I20260806 09:11:35.264585 15308 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39371
I20260806 09:11:35.264915 15308 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:35.265652 15308 heartbeater.cc:511] Master 127.3.69.62:39371 requested a full tablet report, sending...
I20260806 09:11:35.267781 15044 ts_manager.cc:194] Registered new tserver with Master: 64387a9987334a36a5a1d04ebdde95fb (127.3.69.1:39791)
I20260806 09:11:35.269220 15044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:37283
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:11:35.305999 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39371
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:35.644941 15316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:35.645196 15316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:35.645289 15316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:35.652508 15316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:35.652609 15316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:35.652725 15316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:35.662272 15316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:35.663565 15316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:35.664839 15316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:35.677951 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:35.678020 15323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:35.677989 15326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:35.678707 15316 server_base.cc:1034] running on GCE node
I20260806 09:11:35.679476 15316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:35.680503 15316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:35.681712 15316 hybrid_clock.cc:648] HybridClock initialized: now 1786007495681646 us; error 55 us; skew 500 ppm
I20260806 09:11:35.682040 15316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:35 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:35.685349 15316 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:11:35.686808 15316 webserver.cc:460] Webserver started at http://127.3.69.2:38635/ using document root <none> and password file <none>
I20260806 09:11:35.687508 15316 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:35.687623 15316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:35.687889 15316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:35.690919 15316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "5651faf9d32e4d7e800f6ac936bd2b88"
format_stamp: "Formatted at 2026-08-06 09:11:35 on dist-test-slave-w4v5"
server_key: "79bbd007f31895a7e2dc08e23403118c"
server_key_iv: "f7907c2cc57e4538f1d6ecb56c2edd2d"
server_key_version: "encryptionkey@0"
I20260806 09:11:35.691730 15316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "5651faf9d32e4d7e800f6ac936bd2b88"
format_stamp: "Formatted at 2026-08-06 09:11:35 on dist-test-slave-w4v5"
server_key: "79bbd007f31895a7e2dc08e23403118c"
server_key_iv: "f7907c2cc57e4538f1d6ecb56c2edd2d"
server_key_version: "encryptionkey@0"
I20260806 09:11:35.696815 15316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 09:11:35.700182 15333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:35.701582 15316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:11:35.701713 15316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "5651faf9d32e4d7e800f6ac936bd2b88"
format_stamp: "Formatted at 2026-08-06 09:11:35 on dist-test-slave-w4v5"
server_key: "79bbd007f31895a7e2dc08e23403118c"
server_key_iv: "f7907c2cc57e4538f1d6ecb56c2edd2d"
server_key_version: "encryptionkey@0"
I20260806 09:11:35.701890 15316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:35.713598 15316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:35.717247 15316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:35.717489 15316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:35.718335 15316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:35.719621 15316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:35.719696 15316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:35.719776 15316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:35.719825 15316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:35.751936 15316 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:35505
I20260806 09:11:35.751981 15495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:35505 every 8 connection(s)
I20260806 09:11:35.753410 15316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:35.761822 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 15316
I20260806 09:11:35.762006 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:11:35.762463 3348 external_mini_cluster.cc:1428] Setting key 5391fa2dd932bf8dc8f622c81e293ba6
Aug 06 09:11:35 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:35.774788 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:35.756742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:42073 (local address 127.3.69.62:39371)
0806 09:11:35.756933 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:35.756937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:35.758184 (+ 1247us) server_negotiation.cc:407] Connection header received
0806 09:11:35.760146 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:35.760151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:35.760243 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:35.760356 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:35.761843 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:35.762725 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:35.764762 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:35.765198 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:35.765334 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:35.768906 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:35.768911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:35.768915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:35.768949 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:35.771763 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:35.772507 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:35.772515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:35.772523 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:35.772649 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:35.773113 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:35.773117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:35.773119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:35.773380 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:35.773580 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:35.774368 (+ 788us) server_negotiation.cc:299] Negotiation successful
0806 09:11:35.774547 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 09:11:35.776310 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:35.757235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:42073)
0806 09:11:35.758011 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:35.758058 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:35.759877 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:35.760592 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:35.760600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:35.761092 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:35.761669 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:35.761686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:35.762890 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:35.762894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:35.764617 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:35.764629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:35.765433 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:35.765441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:35.765670 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:35.766366 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:35.766390 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:35.768668 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:35.771914 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:35.771923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:35.771940 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:35.772371 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:35.772765 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:35.772770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:35.772774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:35.772965 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:35.773516 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:35.773525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:35.773880 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 09:11:35.774213 (+ 333us) client_negotiation.cc:240] Negotiation successful
0806 09:11:35.774823 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":332,"threads_started":1}
I20260806 09:11:35.777774 15496 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39371
I20260806 09:11:35.778098 15496 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:35.778838 15496 heartbeater.cc:511] Master 127.3.69.62:39371 requested a full tablet report, sending...
I20260806 09:11:35.780459 15044 ts_manager.cc:194] Registered new tserver with Master: 5651faf9d32e4d7e800f6ac936bd2b88 (127.3.69.2:35505)
I20260806 09:11:35.781289 15044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:42073
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:11:35.823510 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39371
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:36.161270 15503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:36.161586 15503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:36.161767 15503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:36.169057 15503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:36.169206 15503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:36.169384 15503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:36.179158 15503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:36.180441 15503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:36.181820 15503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:36.201007 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:36.206449 15511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:36.207270 15516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:36.207712 15503 server_base.cc:1034] running on GCE node
I20260806 09:11:36.208485 15503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:36.209611 15503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:36.210846 15503 hybrid_clock.cc:648] HybridClock initialized: now 1786007496210799 us; error 38 us; skew 500 ppm
I20260806 09:11:36.211236 15503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:36 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007496, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:36.214941 15503 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:11:36.216913 15503 webserver.cc:460] Webserver started at http://127.3.69.3:42957/ using document root <none> and password file <none>
I20260806 09:11:36.217926 15503 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:36.218086 15503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:36.218488 15503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:36.223295 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "8874ff1aaada4bdcbbc686749fba4fb1"
format_stamp: "Formatted at 2026-08-06 09:11:36 on dist-test-slave-w4v5"
server_key: "93a55ab82d79b2e7fbe5f0658aa489a0"
server_key_iv: "44c748b29ac3aef5f77fa3f0495b43e0"
server_key_version: "encryptionkey@0"
I20260806 09:11:36.224503 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "8874ff1aaada4bdcbbc686749fba4fb1"
format_stamp: "Formatted at 2026-08-06 09:11:36 on dist-test-slave-w4v5"
server_key: "93a55ab82d79b2e7fbe5f0658aa489a0"
server_key_iv: "44c748b29ac3aef5f77fa3f0495b43e0"
server_key_version: "encryptionkey@0"
I20260806 09:11:36.232429 15503 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 09:11:36.237679 15527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:36.239607 15503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:11:36.239962 15503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "8874ff1aaada4bdcbbc686749fba4fb1"
format_stamp: "Formatted at 2026-08-06 09:11:36 on dist-test-slave-w4v5"
server_key: "93a55ab82d79b2e7fbe5f0658aa489a0"
server_key_iv: "44c748b29ac3aef5f77fa3f0495b43e0"
server_key_version: "encryptionkey@0"
I20260806 09:11:36.240275 15503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:36.272532 15308 heartbeater.cc:503] Master 127.3.69.62:39371 was elected leader, sending a full tablet report...
I20260806 09:11:36.276329 15503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:36.281221 15503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:36.281567 15503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:36.282613 15503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:36.284225 15503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:36.284303 15503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:36.284471 15503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:36.284596 15503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 09:11:36.324121 15503 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:33273
I20260806 09:11:36.324180 15692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:33273 every 8 connection(s)
I20260806 09:11:36.325486 15503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:36.330704 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 15503
I20260806 09:11:36.330901 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:11:36.331540 3348 external_mini_cluster.cc:1428] Setting key b98f7092075398cdd1cfda4fa08ea38a
Aug 06 09:11:36 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007496, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:11:36.348618 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:36.328625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:47515 (local address 127.3.69.62:39371)
0806 09:11:36.329159 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:36.329164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:36.330333 (+ 1169us) server_negotiation.cc:407] Connection header received
0806 09:11:36.332989 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:36.332995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:36.333086 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:36.333207 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:36.334703 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.335769 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.337370 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.337766 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.337909 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:36.341863 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:36.341869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:36.341872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:36.341906 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:36.344807 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:36.345609 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:36.345616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:36.345620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:36.345810 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:36.346278 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:36.346286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:36.346291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:36.346734 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:36.346915 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:36.347994 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0806 09:11:36.348228 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":286,"threads_started":1}
I20260806 09:11:36.350023 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:36.329278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:47515)
0806 09:11:36.330123 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:36.330201 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:36.332711 (+ 2510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:36.333448 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:36.333457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:36.333964 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:36.334531 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:36.334547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.335921 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.335926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:36.337204 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:36.337216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.338020 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.338027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:36.338253 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:36.339002 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:36.339026 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:36.341451 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:36.344952 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:36.344961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:36.344975 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:36.345485 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:36.345928 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:11:36.345933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:11:36.345937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:11:36.346121 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:11:36.347016 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:36.347028 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:36.347538 (+ 510us) client_negotiation.cc:769] Sending connection context
0806 09:11:36.347885 (+ 347us) client_negotiation.cc:240] Negotiation successful
0806 09:11:36.348488 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":352,"threads_started":1}
I20260806 09:11:36.351583 15694 heartbeater.cc:347] Connected to a master server at 127.3.69.62:39371
I20260806 09:11:36.351910 15694 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:36.352619 15694 heartbeater.cc:511] Master 127.3.69.62:39371 requested a full tablet report, sending...
I20260806 09:11:36.354055 15044 ts_manager.cc:194] Registered new tserver with Master: 8874ff1aaada4bdcbbc686749fba4fb1 (127.3.69.3:33273)
I20260806 09:11:36.354877 15044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:47515
I20260806 09:11:36.361194 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab.
Entry for principal oryx/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab.
WARNING: no policy specified for oryx/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab.
Entry for principal oryx/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab.
WARNING: no policy specified for oryx/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab.
Entry for principal oryx/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab.
I20260806 09:11:36.525398 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 15126
Aug 06 09:11:36 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:11:36.542639 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:36.526643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.3.69.1:39791)
0806 09:11:36.527350 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:36.527357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:36.527386 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:11:36.527452 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:36.527458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:36.527675 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:36.527820 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:36.529398 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.530567 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.532001 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.532605 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.532773 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:36.536839 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:36.536851 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:36.536866 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:36.536905 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:36.539967 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:36.540764 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:36.540771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:36.540773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:36.540911 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:36.541276 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:36.541279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:36.541281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:36.541614 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:36.541872 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:36.542123 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 09:11:36.542332 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":350,"threads_started":1}
I20260806 09:11:36.544257 15302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:36.631680 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 15126
I20260806 09:11:36.655409 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 15316
Aug 06 09:11:36 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:11:36.672547 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:36.656542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.3.69.2:35505)
0806 09:11:36.657218 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:36.657225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:36.657255 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:11:36.657321 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:36.657327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:36.657560 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:36.657701 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:36.658930 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.660196 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.661429 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.662044 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.662163 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:36.666430 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:36.666442 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:36.666457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:36.666495 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:36.669809 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:36.670488 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:36.670496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:36.670498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:36.670671 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:36.671020 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:36.671024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:36.671026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:36.671451 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:36.671779 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:36.671931 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 09:11:36.672138 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":331,"threads_started":1}
I20260806 09:11:36.673882 15487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:36.755690 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 15316
I20260806 09:11:36.784384 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 15503
Aug 06 09:11:36 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:11:36.801712 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:36.785587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.3.69.3:33273)
0806 09:11:36.786120 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:36.786127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:36.786158 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:11:36.786219 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:36.786225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:36.786442 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:36.786650 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:36.788068 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.789104 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.790357 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:36.791135 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:36.791291 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:36.795735 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:36.795746 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:36.795760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:36.795797 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:36.798740 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:36.799505 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:36.799513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:36.799515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:36.799645 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:36.800041 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:36.800046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:36.800048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:36.800396 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:36.800724 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:36.801029 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 09:11:36.801311 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":256,"threads_started":1}
I20260806 09:11:36.803016 15686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:36.886286 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 15503
I20260806 09:11:36.911324 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 14997
I20260806 09:11:36.912573 15084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:37.022611 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 14997
I20260806 09:11:37.048611 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39371
--webserver_interface=127.3.69.62
--webserver_port=37277
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:11:37.386727 15721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:37.387022 15721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:37.387205 15721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:37.394636 15721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:37.394737 15721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:37.394824 15721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:37.394933 15721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:37.395016 15721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:37.405028 15721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:39371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=37277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:37.406322 15721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:37.407703 15721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:37.420054 15732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:37.420168 15728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:37.420138 15730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:37.421144 15721 server_base.cc:1034] running on GCE node
I20260806 09:11:37.421975 15721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:37.423451 15721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:37.424691 15721 hybrid_clock.cc:648] HybridClock initialized: now 1786007497424628 us; error 55 us; skew 500 ppm
I20260806 09:11:37.425015 15721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:37 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007497, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:37.428290 15721 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:11:37.429899 15721 webserver.cc:460] Webserver started at http://127.3.69.62:37277/ using document root <none> and password file <none>
I20260806 09:11:37.430668 15721 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:37.430845 15721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:37.436343 15721 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260806 09:11:37.439201 15741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:37.440547 15721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:37.440753 15721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "db7e165a38e744ca94124452b9ef9fc6"
format_stamp: "Formatted at 2026-08-06 09:11:34 on dist-test-slave-w4v5"
server_key: "38d4d82f15d9636c8746b70be3c5d091"
server_key_iv: "62cee980d2b5cf2f2862a3db16c61759"
server_key_version: "encryptionkey@0"
I20260806 09:11:37.441715 15721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:37.475303 15721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:37.479475 15721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:37.479781 15721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:37.498853 15721 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:39371
I20260806 09:11:37.498889 15813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:39371 every 8 connection(s)
I20260806 09:11:37.500448 15721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:37.505735 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 15721
I20260806 09:11:37.506580 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:39791
--local_ip_for_outbound_sockets=127.3.69.1
--tserver_master_addrs=127.3.69.62:39371
--webserver_port=45493
--webserver_interface=127.3.69.1
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.69.1
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:11:37.507791 15814 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:11:37.511042 15814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Bootstrap starting.
I20260806 09:11:37.530175 15814 log.cc:824] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:37.539251 15814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:11:37.539795 15814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Bootstrap complete.
I20260806 09:11:37.549183 15814 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } }
I20260806 09:11:37.550026 15814 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7e165a38e744ca94124452b9ef9fc6, State: Initialized, Role: FOLLOWER
I20260806 09:11:37.550627 15814 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } }
I20260806 09:11:37.550881 15814 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:37.551012 15814 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:37.551230 15814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:11:37.554879 15814 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } }
I20260806 09:11:37.555394 15814 leader_election.cc:302] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7e165a38e744ca94124452b9ef9fc6; no voters:
I20260806 09:11:37.556052 15814 leader_election.cc:288] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:11:37.556193 15818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:11:37.557425 15818 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [term 2 LEADER]: Becoming Leader. State: Replica: db7e165a38e744ca94124452b9ef9fc6, State: Running, Role: LEADER
I20260806 09:11:37.558035 15818 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } }
I20260806 09:11:37.558329 15814 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:37.561563 15820 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7e165a38e744ca94124452b9ef9fc6. Latest consensus state: current_term: 2 leader_uuid: "db7e165a38e744ca94124452b9ef9fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } } }
I20260806 09:11:37.561787 15819 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "db7e165a38e744ca94124452b9ef9fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7e165a38e744ca94124452b9ef9fc6" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39371 } } }
I20260806 09:11:37.562493 15820 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:37.562547 15819 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6 [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:37.573933 15839 catalog_manager.cc:1259] Loaded cluster ID: ed96e926a6be4f3695b2259dec13d8be
I20260806 09:11:37.574107 15839 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: loading cluster ID for follower catalog manager: success
I20260806 09:11:37.578181 15839 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: acquiring CA information for follower catalog manager: success
I20260806 09:11:37.580544 15839 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:11:37.580766 15827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:37.582571 15827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:37.583494 15827 catalog_manager.cc:1259] Loaded cluster ID: ed96e926a6be4f3695b2259dec13d8be
I20260806 09:11:37.583613 15827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:37.584810 15827 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:37.585685 15827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db7e165a38e744ca94124452b9ef9fc6: Loaded TSK: 0
I20260806 09:11:37.586408 15827 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:11:37.881958 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:37.882238 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:37.882383 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:37.889804 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:37.889912 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:37.890076 15816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:37.899796 15816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.1.keytab
--principal=oryx/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:39791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=45493
--tserver_master_addrs=127.3.69.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:37.901145 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:37.902465 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:37.917400 15850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:37.917765 15849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:37.917388 15852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:37.918421 15816 server_base.cc:1034] running on GCE node
I20260806 09:11:37.919139 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:37.920238 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:37.921487 15816 hybrid_clock.cc:648] HybridClock initialized: now 1786007497921423 us; error 51 us; skew 500 ppm
I20260806 09:11:37.921835 15816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:37 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007497, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:37.925616 15816 init.cc:377] Logged in from keytab as oryx/127.3.69.1@KRBTEST.COM (short username oryx)
I20260806 09:11:37.927232 15816 webserver.cc:460] Webserver started at http://127.3.69.1:45493/ using document root <none> and password file <none>
I20260806 09:11:37.927907 15816 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:37.928058 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:37.933601 15816 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:11:37.936436 15866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:37.937850 15816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:11:37.938028 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "64387a9987334a36a5a1d04ebdde95fb"
format_stamp: "Formatted at 2026-08-06 09:11:35 on dist-test-slave-w4v5"
server_key: "6c53a45f9965cdcffe266bfe32f61940"
server_key_iv: "48d3ca04410e33676bd765cc72b2b457"
server_key_version: "encryptionkey@0"
I20260806 09:11:37.939159 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:37.950577 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:37.954869 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:37.955224 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:37.956224 15816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:37.957676 15816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:37.957813 15816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:37.957940 15816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:37.958029 15816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:37.992234 15816 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:39791
I20260806 09:11:37.992326 16037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:39791 every 8 connection(s)
I20260806 09:11:37.993736 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:37.994261 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 15816
I20260806 09:11:37.995184 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:35505
--local_ip_for_outbound_sockets=127.3.69.2
--tserver_master_addrs=127.3.69.62:39371
--webserver_port=38635
--webserver_interface=127.3.69.2
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.69.2
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:38.012626 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:37.997094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:40217)
0806 09:11:37.997944 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:37.997997 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:37.999995 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.001347 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.001360 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.001973 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.002485 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.002500 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.004145 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.004151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.005352 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.005365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.006381 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.006388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.006653 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.007417 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.007440 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.011310 (+ 3870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":354,"threads_started":1}
W20260806 09:11:38.012656 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:37.997145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:40217 (local address 127.3.69.62:39371)
0806 09:11:37.999593 (+ 2448us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:37.999609 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:37.999662 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:11:38.000554 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.000566 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.000865 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.001220 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.002681 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.003977 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.005504 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.006128 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.006341 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.011724 (+ 5383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:40217: BlockingRecv error: recv got EOF from 127.3.69.1:40217 (error 108)
Metrics: {"server-negotiator.queue_time_us":2218,"thread_start_us":1979,"threads_started":1}
W20260806 09:11:38.013296 16038 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:39371 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.69.62:39371: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:38.021741 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.013894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:44907)
0806 09:11:38.014059 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:38.014079 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:38.014196 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.014585 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.014590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.014943 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.015127 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.015144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.016103 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.016109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.016998 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.017009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.017707 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.017713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.017885 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.018469 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.018486 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.021449 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 09:11:38.022145 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:38.014127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:44907 (local address 127.3.69.62:39371)
0806 09:11:38.014329 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.014335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.014357 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:11:38.014399 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.014402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.014454 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.014557 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.015269 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.015964 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.017144 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.017556 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.017686 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.021636 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:44907: BlockingRecv error: recv got EOF from 127.3.69.1:44907 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:38.030014 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.022380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:40197)
0806 09:11:38.022535 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:38.022560 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:38.022673 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.023256 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.023262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.023557 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.023698 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.023705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.024677 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.024682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.025563 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.025573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.026245 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.026251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.026410 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.026986 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.027003 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.029713 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 09:11:38.030282 16038 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:11:38.030441 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:38.022775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:40197 (local address 127.3.69.62:39371)
0806 09:11:38.022940 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.022944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.022957 (+ 13us) server_negotiation.cc:407] Connection header received
0806 09:11:38.022994 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.022997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.023063 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.023222 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.023831 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.024534 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.025720 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.026094 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.026224 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.029903 (+ 3679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:40197: BlockingRecv error: recv got EOF from 127.3.69.1:40197 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 09:11:38.332944 16043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:38.333271 16043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:38.333439 16043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:38.340677 16043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:38.340840 16043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:38.341012 16043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:38.350677 16043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.2.keytab
--principal=oryx/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:35505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=38635
--tserver_master_addrs=127.3.69.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:38.352034 16043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:38.353329 16043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:38.366915 16049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:38.366907 16050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:38.367261 16053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:38.368144 16043 server_base.cc:1034] running on GCE node
I20260806 09:11:38.368826 16043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:38.369810 16043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:38.371136 16043 hybrid_clock.cc:648] HybridClock initialized: now 1786007498371009 us; error 52 us; skew 500 ppm
I20260806 09:11:38.371470 16043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007498, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:38.374779 16043 init.cc:377] Logged in from keytab as oryx/127.3.69.2@KRBTEST.COM (short username oryx)
I20260806 09:11:38.376344 16043 webserver.cc:460] Webserver started at http://127.3.69.2:38635/ using document root <none> and password file <none>
I20260806 09:11:38.377025 16043 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:38.377197 16043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:38.382414 16043 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 09:11:38.385071 16062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:38.386406 16043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:11:38.386612 16043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "5651faf9d32e4d7e800f6ac936bd2b88"
format_stamp: "Formatted at 2026-08-06 09:11:35 on dist-test-slave-w4v5"
server_key: "79bbd007f31895a7e2dc08e23403118c"
server_key_iv: "f7907c2cc57e4538f1d6ecb56c2edd2d"
server_key_version: "encryptionkey@0"
I20260806 09:11:38.387687 16043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:38.419162 16043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:38.423658 16043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:38.423959 16043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:38.425004 16043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:38.426373 16043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:38.426502 16043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:38.426651 16043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:38.426743 16043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:38.461927 16043 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:35505
I20260806 09:11:38.462033 16216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:35505 every 8 connection(s)
I20260806 09:11:38.463474 16043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:38.463946 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 16043
I20260806 09:11:38.464834 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:33273
--local_ip_for_outbound_sockets=127.3.69.3
--tserver_master_addrs=127.3.69.62:39371
--webserver_port=42957
--webserver_interface=127.3.69.3
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.69.3
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:11:38.480075 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:38.467394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:44941 (local address 127.3.69.62:39371)
0806 09:11:38.467624 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.467631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.468027 (+ 396us) server_negotiation.cc:407] Connection header received
0806 09:11:38.469901 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.469907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.470002 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.470124 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.471704 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.472435 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.473975 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.474650 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.474794 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.479742 (+ 4948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:44941: BlockingRecv error: recv got EOF from 127.3.69.2:44941 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:11:38.480363 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.466968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:44941)
0806 09:11:38.467864 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:38.467902 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:38.469634 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.470362 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.470369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.470890 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.471514 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.471531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.472618 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.472622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.473829 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.473841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.474873 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.474879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.475200 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.475950 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.475972 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.479484 (+ 3512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":369,"threads_started":1}
W20260806 09:11:38.481164 16217 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:39371 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.69.62:39371: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:38.490754 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.481739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:48861)
0806 09:11:38.481951 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:38.481973 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:38.482087 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.482420 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.482427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.482859 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.483143 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.483158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.484112 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.484117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.485019 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.485030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.485774 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.485783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.486027 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.486620 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.486649 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.490391 (+ 3742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 09:11:38.490792 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:38.481883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:48861 (local address 127.3.69.62:39371)
0806 09:11:38.482043 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.482047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.482063 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:11:38.482199 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.482202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.482266 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.482347 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.483279 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.483961 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.485181 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.485594 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.485738 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.490616 (+ 4878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:48861: BlockingRecv error: recv got EOF from 127.3.69.2:48861 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:38.499909 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.491596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:56747)
0806 09:11:38.491798 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:38.491821 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:38.491928 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.492266 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.492270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.492632 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.492775 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.492783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.494297 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.494300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.495299 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.495311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.496196 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.496201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.496320 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.496889 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.496906 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.499615 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 09:11:38.500065 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:38.491722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:56747 (local address 127.3.69.62:39371)
0806 09:11:38.491853 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.491857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.491872 (+ 15us) server_negotiation.cc:407] Connection header received
0806 09:11:38.492056 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.492061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.492132 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.492238 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.492956 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.494155 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.495429 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.496065 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.496189 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.499801 (+ 3612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:56747: BlockingRecv error: recv got EOF from 127.3.69.2:56747 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260806 09:11:38.500427 16217 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:11:38.812002 16221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:38.812285 16221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:38.812381 16221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:38.819396 16221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:38.819504 16221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:38.819622 16221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:38.829275 16221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/krb5kdc/oryx_127.3.69.3.keytab
--principal=oryx/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:33273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=42957
--tserver_master_addrs=127.3.69.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:38.830622 16221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:38.831998 16221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:38.845331 16231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:38.845597 16232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:38.845620 16234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:38.846755 16221 server_base.cc:1034] running on GCE node
I20260806 09:11:38.847433 16221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:38.848877 16221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:38.850143 16221 hybrid_clock.cc:648] HybridClock initialized: now 1786007498850087 us; error 28 us; skew 500 ppm
I20260806 09:11:38.850544 16221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007498, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:11:38.853816 16221 init.cc:377] Logged in from keytab as oryx/127.3.69.3@KRBTEST.COM (short username oryx)
I20260806 09:11:38.855273 16221 webserver.cc:460] Webserver started at http://127.3.69.3:42957/ using document root <none> and password file <none>
I20260806 09:11:38.855968 16221 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:38.856112 16221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:38.861416 16221 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 09:11:38.864207 16243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:38.865525 16221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:11:38.865751 16221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "8874ff1aaada4bdcbbc686749fba4fb1"
format_stamp: "Formatted at 2026-08-06 09:11:36 on dist-test-slave-w4v5"
server_key: "93a55ab82d79b2e7fbe5f0658aa489a0"
server_key_iv: "44c748b29ac3aef5f77fa3f0495b43e0"
server_key_version: "encryptionkey@0"
I20260806 09:11:38.866701 16221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:38.885562 16221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:38.889501 16221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:38.889782 16221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:38.890820 16221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:38.892243 16221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:38.892390 16221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:38.892532 16221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:38.892627 16221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:38.926990 16221 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:33273
I20260806 09:11:38.927048 16407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:33273 every 8 connection(s)
I20260806 09:11:38.928474 16221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:38.933580 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 16221
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:11:38.945801 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:38.931188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:37857 (local address 127.3.69.62:39371)
0806 09:11:38.931368 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.931373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.932540 (+ 1167us) server_negotiation.cc:407] Connection header received
0806 09:11:38.934851 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.934860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.934987 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.935202 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.937275 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.937967 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.939707 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.940110 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.940248 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.945507 (+ 5259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:37857: BlockingRecv error: recv got EOF from 127.3.69.3:37857 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 09:11:38.946491 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.931652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:37857)
0806 09:11:38.932370 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:38.932406 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:38.934492 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.935455 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.935469 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.936243 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.937098 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.937126 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.938128 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.938133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.939502 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.939514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.940354 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.940361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.940590 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.941277 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.941314 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.945320 (+ 4006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":285,"threads_started":1}
W20260806 09:11:38.947372 16408 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:39371 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.69.62:39371: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
I20260806 09:11:38.950883 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.934674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.3.69.62:39371)
0806 09:11:38.935277 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.935283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.935869 (+ 586us) server_negotiation.cc:407] Connection header received
0806 09:11:38.936040 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.936043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.936101 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.936193 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.937986 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.938801 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.940270 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.940705 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.940838 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.943572 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:38.943582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:38.943598 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:38.943632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:38.947495 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:38.948404 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:38.948411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:38.948417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:38.948609 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:11:38.949002 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:11:38.949006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:11:38.949008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:11:38.949470 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:38.949848 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:38.950245 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 09:11:38.950607 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":497,"thread_start_us":316,"threads_started":1}
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:38.956087 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.948005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:42691)
0806 09:11:38.948166 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:38.948187 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:38.948287 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.948615 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.948621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.948951 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.949108 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.949116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.950168 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.950174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.951286 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.951298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.952025 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.952031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.952215 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.952765 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.952782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.955891 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 09:11:38.956279 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:38.948112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:42691 (local address 127.3.69.62:39371)
0806 09:11:38.948270 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.948275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.948293 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:11:38.948404 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.948409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.948479 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.948585 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.949250 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.950020 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.951431 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.951876 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.952014 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.956052 (+ 4038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:42691: BlockingRecv error: recv got EOF from 127.3.69.3:42691 (error 108)
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":57}
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:38 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:38.964267 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:38.956908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:34973)
0806 09:11:38.957072 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:38.957093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:38.957227 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:38.957584 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:38.957590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:38.957895 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:38.958027 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.958034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.959034 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.959040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.959997 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:38.960008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.960682 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.960688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:38.960856 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.961337 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:38.961354 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:38.964048 (+ 2694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:11:38.964493 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:38.957127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:34973 (local address 127.3.69.62:39371)
0806 09:11:38.957360 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:38.957364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:38.957377 (+ 13us) server_negotiation.cc:407] Connection header received
0806 09:11:38.957410 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:38.957413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:38.957459 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:38.957557 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:38.958168 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.958897 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.960147 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:38.960537 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:38.960667 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:38.964228 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:34973: BlockingRecv error: recv got EOF from 127.3.69.3:34973 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260806 09:11:38.964557 16408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 09:11:39 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:39 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:39.040181 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:39.030887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:36501)
0806 09:11:39.031589 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:39.031606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:39.031759 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:39.032123 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:39.032127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:39.032630 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:39.032830 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:39.032839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.033843 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.033847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:39.034809 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:39.034820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.035607 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.035612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:39.035733 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:39.036321 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:39.036343 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:39.039837 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":402,"threads_started":1}
W20260806 09:11:39.040277 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:39.030992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:36501 (local address 127.3.69.62:39371)
0806 09:11:39.031220 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:39.031225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:39.031686 (+ 461us) server_negotiation.cc:407] Connection header received
0806 09:11:39.031875 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:39.031879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:39.031942 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:39.032035 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:39.032974 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.033692 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.034935 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.035410 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.035506 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:39.040049 (+ 4543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:36501: BlockingRecv error: recv got EOF from 127.3.69.1:36501 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 06 09:11:39 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:39 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:39.509883 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:39.501046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:45541)
0806 09:11:39.501601 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:39.501619 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:39.501734 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:39.502080 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:39.502083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:39.502559 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:39.502754 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:39.502763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.504234 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.504238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:39.505233 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:39.505246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.505993 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.505998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:39.506131 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:39.506763 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:39.506780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:39.509665 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":298,"threads_started":1}
W20260806 09:11:39.510124 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:39.501130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:45541 (local address 127.3.69.62:39371)
0806 09:11:39.501328 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:39.501335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:39.501714 (+ 379us) server_negotiation.cc:407] Connection header received
0806 09:11:39.501845 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:39.501851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:39.501928 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:39.502035 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:39.502906 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.504063 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.505405 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.505835 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.505936 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:39.509855 (+ 3919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:45541: BlockingRecv error: recv got EOF from 127.3.69.2:45541 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 09:11:39 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:39 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:39.974649 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:39.965166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:35893)
0806 09:11:39.965737 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:39.965753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:39.965931 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:39.966335 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:39.966339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:39.966914 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:39.967161 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:39.967171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.968247 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.968251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:39.969220 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:39.969232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.970104 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.970108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:39.970236 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:39.970961 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:39.970980 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:39.974265 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":302,"threads_started":1}
W20260806 09:11:39.974761 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:39.965261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:35893 (local address 127.3.69.62:39371)
0806 09:11:39.965529 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:39.965537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:39.965844 (+ 307us) server_negotiation.cc:407] Connection header received
0806 09:11:39.966069 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:39.966074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:39.966154 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:39.966289 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:39.967315 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.968098 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.969377 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:39.969925 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:39.970030 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:39.974510 (+ 4480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:35893: BlockingRecv error: recv got EOF from 127.3.69.3:35893 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 06 09:11:40 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:40 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:40.051402 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:40.041347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:59255)
0806 09:11:40.043007 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:40.043025 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:40.043258 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:40.043602 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:40.043607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:40.044105 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:40.044258 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:40.044266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.045386 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.045392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:40.046497 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:40.046510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.047317 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.047323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:40.047494 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:40.048147 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:40.048166 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:40.051147 (+ 2981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1543,"thread_start_us":1417,"threads_started":1}
W20260806 09:11:40.051569 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:40.041439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:59255 (local address 127.3.69.62:39371)
0806 09:11:40.041642 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:40.041651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:40.043179 (+ 1528us) server_negotiation.cc:407] Connection header received
0806 09:11:40.043393 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:40.043398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:40.043462 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:40.043578 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:40.044407 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.045244 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.046650 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.047164 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.047303 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:40.051342 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:59255: BlockingRecv error: recv got EOF from 127.3.69.1:59255 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 09:11:40 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:40 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:40.520025 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:40.510780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:54905)
0806 09:11:40.511418 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:40.511436 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:40.511624 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:40.511966 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:40.511971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:40.512425 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:40.512601 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:40.512610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.513637 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.513644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:40.514598 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:40.514618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.515695 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.515703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:40.515856 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:40.516607 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:40.516634 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:40.519765 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":367,"threads_started":1}
W20260806 09:11:40.520207 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:40.510940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:54905 (local address 127.3.69.62:39371)
0806 09:11:40.511228 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:40.511236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:40.511526 (+ 290us) server_negotiation.cc:407] Connection header received
0806 09:11:40.511755 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:40.511764 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:40.511827 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:40.511933 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:40.512747 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.513494 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.514799 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.515468 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.515620 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:40.519963 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:54905: BlockingRecv error: recv got EOF from 127.3.69.2:54905 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 09:11:40 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:40 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:40.995234 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:40.976207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:34745)
0806 09:11:40.977620 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:40.977654 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:40.977905 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:40.978787 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:40.978814 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:40.979737 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:40.980205 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:40.980224 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.981866 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.981870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:40.983651 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:40.983678 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.985107 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.985113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:40.985465 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:40.986710 (+ 1245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:40.986745 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:40.994457 (+ 7712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1238,"thread_start_us":868,"threads_started":1}
W20260806 09:11:40.995244 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:40.976424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:34745 (local address 127.3.69.62:39371)
0806 09:11:40.976909 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:40.976927 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:40.977784 (+ 857us) server_negotiation.cc:407] Connection header received
0806 09:11:40.978205 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:40.978217 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:40.978322 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:40.978677 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:40.980442 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.981676 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.984002 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:40.984897 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:40.985053 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:40.994693 (+ 9640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:34745: BlockingRecv error: recv got EOF from 127.3.69.3:34745 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 06 09:11:41 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:41 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:41.072400 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:41.052965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:36135)
0806 09:11:41.053837 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:41.053871 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:41.054282 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:41.054988 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:41.055000 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:41.055705 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:41.056132 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:41.056150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:41.058297 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:41.058304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:41.060896 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:41.060934 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:41.062832 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:41.062866 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:41.063403 (+ 537us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:41.064565 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:41.064620 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:41.071358 (+ 6738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":554,"threads_started":1}
W20260806 09:11:41.072773 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:41.053073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:36135 (local address 127.3.69.62:39371)
0806 09:11:41.053589 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:41.053606 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:41.054064 (+ 458us) server_negotiation.cc:407] Connection header received
0806 09:11:41.054470 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:41.054481 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:41.054634 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:41.054930 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:41.056351 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:41.057824 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:41.061532 (+ 3708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:41.062206 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:41.062787 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:41.072108 (+ 9321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:36135: BlockingRecv error: recv got EOF from 127.3.69.1:36135 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Aug 06 09:11:41 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:41 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:41.590706 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:41.526603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:47119)
0806 09:11:41.529707 (+ 3104us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:41.529767 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:41.530166 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:41.531667 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:41.531690 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:41.534037 (+ 2347us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:41.535235 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:41.535284 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:41.548116 (+ 12832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:41.548131 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:41.554946 (+ 6815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:41.555042 (+ 96us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:41.560026 (+ 4984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:41.560110 (+ 84us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:41.560890 (+ 780us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:41.564072 (+ 3182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:41.564175 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:41.587380 (+ 23205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2594,"thread_start_us":2134,"threads_started":1}
W20260806 09:11:41.593048 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:41.527623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:47119 (local address 127.3.69.62:39371)
0806 09:11:41.528682 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:41.528805 (+ 123us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:41.529985 (+ 1180us) server_negotiation.cc:407] Connection header received
0806 09:11:41.530553 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:41.530643 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:41.530978 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:41.531662 (+ 684us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:41.536009 (+ 4347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:41.547273 (+ 11264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:41.555811 (+ 8538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:41.558873 (+ 3062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:41.560019 (+ 1146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:41.589763 (+ 29744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:47119: BlockingRecv error: recv got EOF from 127.3.69.2:47119 (error 108)
Metrics: {"server-negotiator.queue_time_us":444}
Aug 06 09:11:42 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:42 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:42.034516 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:41.997721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:39613)
0806 09:11:41.999002 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:41.999044 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:41.999552 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:42.000691 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:42.000710 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:42.001988 (+ 1278us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:42.002692 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:42.002721 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.006458 (+ 3737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.006464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:42.008454 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:42.008470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.010965 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.010974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:42.011667 (+ 693us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:42.013496 (+ 1829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:42.013573 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:42.033945 (+ 20372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":599,"threads_started":1}
W20260806 09:11:42.034694 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:41.997858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:39613 (local address 127.3.69.62:39371)
0806 09:11:41.998360 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:41.998381 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:41.999222 (+ 841us) server_negotiation.cc:407] Connection header received
0806 09:11:41.999907 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:41.999917 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:42.000235 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:42.000396 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:42.002973 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.006165 (+ 3192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.008874 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.010661 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.010950 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:42.034238 (+ 23288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:39613: BlockingRecv error: recv got EOF from 127.3.69.3:39613 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Aug 06 09:11:42 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:42 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:42.087473 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:42.074267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:33817)
0806 09:11:42.074982 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:42.075010 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:42.075276 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:42.075772 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:42.075776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:42.076433 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:42.076702 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:42.076713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.078203 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.078207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:42.079891 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:42.079906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.080908 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.080914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:42.081152 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:42.082247 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:42.082286 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:42.086965 (+ 4679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":413,"threads_started":1}
W20260806 09:11:42.087744 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:42.074304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:33817 (local address 127.3.69.62:39371)
0806 09:11:42.074661 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:42.074668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:42.075171 (+ 503us) server_negotiation.cc:407] Connection header received
0806 09:11:42.075466 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:42.075471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:42.075577 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:42.075721 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:42.076890 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.078044 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.080047 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.080695 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.080870 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:42.087379 (+ 6509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:33817: BlockingRecv error: recv got EOF from 127.3.69.1:33817 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 06 09:11:42 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:42 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:42.619788 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:42.597481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:57699)
0806 09:11:42.598589 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:42.598625 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:42.598912 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:42.599916 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:42.599935 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:42.600732 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:42.601253 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:42.601321 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.604008 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.604014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:42.605895 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:42.605916 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.607490 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.607502 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:42.607749 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:42.612138 (+ 4389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:42.612238 (+ 100us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:42.618843 (+ 6605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":497,"threads_started":1}
W20260806 09:11:42.620138 16433 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:42.597571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:57699 (local address 127.3.69.62:39371)
0806 09:11:42.598769 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:42.598784 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:42.598816 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:11:42.599254 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:42.599262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:42.599554 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:42.599745 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:42.601596 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.603767 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.606180 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:42.607234 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:42.607457 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:42.619471 (+ 12014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:57699: BlockingRecv error: recv got EOF from 127.3.69.2:57699 (error 108)
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":601,"threads_started":1}
Aug 06 09:11:43 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:43 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:43.044570 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:43.035815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:51391)
0806 09:11:43.036401 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:43.036417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:43.036591 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:43.036982 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:43.036988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:43.037412 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:43.037574 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:43.037582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.038658 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.038664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:43.039787 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:43.039799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.040477 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.040482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:43.040669 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:43.041295 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:43.041313 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:43.044331 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":338,"threads_started":1}
W20260806 09:11:43.044762 16433 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:43.035862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:51391 (local address 127.3.69.62:39371)
0806 09:11:43.036039 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:43.036045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:43.036506 (+ 461us) server_negotiation.cc:407] Connection header received
0806 09:11:43.036735 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:43.036745 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:43.036835 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:43.036938 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:43.037714 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.038515 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.039927 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.040334 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.040466 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:43.044516 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:51391: BlockingRecv error: recv got EOF from 127.3.69.3:51391 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 09:11:43 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:43 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:43.097671 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:43.088757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:34805)
0806 09:11:43.089268 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:43.089284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:43.089435 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:43.089768 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:43.089772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:43.090310 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:43.090501 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:43.090510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.091768 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.091775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:43.092809 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:43.092820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.093508 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.093513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:43.093701 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:43.094277 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:43.094294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:43.097297 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":249,"threads_started":1}
W20260806 09:11:43.097743 16433 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:43.088833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:34805 (local address 127.3.69.62:39371)
0806 09:11:43.089032 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:43.089041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:43.089368 (+ 327us) server_negotiation.cc:407] Connection header received
0806 09:11:43.089554 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:43.089564 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:43.089638 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:43.089740 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:43.090651 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.091636 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.092952 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.093366 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.093498 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:43.097490 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:34805: BlockingRecv error: recv got EOF from 127.3.69.1:34805 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 09:11:43 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:43 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:43.629806 16438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:43.620998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:36419)
0806 09:11:43.621566 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:43.621585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:43.621688 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:43.622071 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:43.622076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:43.622568 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:43.622788 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:43.622798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.624170 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.624174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:43.625146 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:43.625159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.625819 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.625824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:43.625946 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:43.626523 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:43.626539 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:43.629446 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":287,"threads_started":1}
W20260806 09:11:43.629940 16441 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:43.621213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:36419 (local address 127.3.69.62:39371)
0806 09:11:43.621708 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:43.621716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:43.621740 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:11:43.621799 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:43.621804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:43.621893 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:43.622017 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:43.622972 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.624023 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.625267 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:43.625696 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:43.625789 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:43.629658 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:36419: BlockingRecv error: recv got EOF from 127.3.69.2:36419 (error 108)
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":276,"threads_started":1}
Aug 06 09:11:44 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:44 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:44.055181 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:44.045722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:59057)
0806 09:11:44.046278 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:44.046294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:44.046407 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:44.046798 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:44.046802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:44.047315 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:44.047496 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:44.047506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.048587 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.048591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:44.049691 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:44.049704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.050493 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.050498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:44.050678 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:44.051362 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:44.051381 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:44.054872 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":264,"threads_started":1}
W20260806 09:11:44.055348 16441 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:44.045834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:59057 (local address 127.3.69.62:39371)
0806 09:11:44.046058 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:44.046065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:44.046372 (+ 307us) server_negotiation.cc:407] Connection header received
0806 09:11:44.046549 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:44.046552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:44.046617 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:44.046702 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:44.047655 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.048429 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.049837 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.050332 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.050481 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:44.055140 (+ 4659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:59057: BlockingRecv error: recv got EOF from 127.3.69.3:59057 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 09:11:44 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:44 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:44.107820 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:44.098784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:59723)
0806 09:11:44.099366 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:44.099382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:44.099497 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:44.099841 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:44.099848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:44.100412 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:44.100665 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:44.100677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.101747 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.101752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:44.102756 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:44.102768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.103577 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.103584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:44.103805 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:44.104438 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:44.104456 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:44.107591 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":351,"threads_started":1}
W20260806 09:11:44.108031 16441 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:44.098821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:59723 (local address 127.3.69.62:39371)
0806 09:11:44.099015 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:44.099020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:44.099455 (+ 435us) server_negotiation.cc:407] Connection header received
0806 09:11:44.099596 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:44.099600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:44.099679 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:44.099797 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:44.100813 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.101601 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.102905 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.103402 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.103529 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:44.107773 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:59723: BlockingRecv error: recv got EOF from 127.3.69.1:59723 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 09:11:44 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:44 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:44.639737 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:44.630734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:60631)
0806 09:11:44.631323 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:44.631340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:44.631451 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:44.631774 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:44.631777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:44.632239 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:44.632392 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:44.632400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.633644 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.633648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:44.634542 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:44.634554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.635302 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.635306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:44.635430 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:44.636089 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:44.636106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:44.639414 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":371,"threads_started":1}
W20260806 09:11:44.639946 16445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:44.630806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:60631 (local address 127.3.69.62:39371)
0806 09:11:44.631286 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:44.631291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:44.631406 (+ 115us) server_negotiation.cc:407] Connection header received
0806 09:11:44.631583 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:44.631587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:44.631649 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:44.631735 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:44.632584 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.633475 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.634663 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:44.635144 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:44.635232 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:44.639650 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:60631: BlockingRecv error: recv got EOF from 127.3.69.2:60631 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
Aug 06 09:11:45 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:45 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:45.064921 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:45.056082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:59647)
0806 09:11:45.056683 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:45.056700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:45.056816 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:45.057112 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:45.057116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:45.057587 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:45.057765 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:45.057774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.058914 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.058920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:45.060113 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:45.060146 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.060888 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.060892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:45.061021 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:45.061671 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:45.061690 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:45.064690 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":334,"threads_started":1}
W20260806 09:11:45.065261 16445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:45.056158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:59647 (local address 127.3.69.62:39371)
0806 09:11:45.056321 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:45.056326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:45.056771 (+ 445us) server_negotiation.cc:407] Connection header received
0806 09:11:45.056927 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:45.056931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:45.056996 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:45.057095 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:45.057889 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.058749 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.060288 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.060734 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.060843 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:45.064908 (+ 4065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:59647: BlockingRecv error: recv got EOF from 127.3.69.3:59647 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 09:11:45 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:45 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:45.118616 16452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:45.108831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:45339)
0806 09:11:45.109631 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:45.109648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:45.109776 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:45.110158 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:45.110162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:45.110606 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:45.110791 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:45.110800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.112274 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.112279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:45.113291 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:45.113304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.114105 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.114113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:45.114322 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:45.115025 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:45.115044 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:45.118374 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":273,"threads_started":1}
W20260806 09:11:45.118933 16445 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:45.108925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:45339 (local address 127.3.69.62:39371)
0806 09:11:45.109199 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:45.109208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:45.109773 (+ 565us) server_negotiation.cc:407] Connection header received
0806 09:11:45.109888 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:45.109893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:45.109963 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:45.110051 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:45.110965 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.112108 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.113451 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.113918 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.114089 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:45.118588 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:45339: BlockingRecv error: recv got EOF from 127.3.69.1:45339 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 09:11:45 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:45 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:45.650876 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:45.640830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:37845)
0806 09:11:45.641485 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:45.641502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:45.641632 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:45.642075 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:45.642080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:45.642665 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:45.642850 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:45.642859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.644487 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.644503 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:45.645632 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:45.645647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.646453 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.646458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:45.646656 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:45.647335 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:45.647353 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:45.650509 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":293,"threads_started":1}
W20260806 09:11:45.651230 16455 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:45.640954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:37845 (local address 127.3.69.62:39371)
0806 09:11:45.641456 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:45.641460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:45.641621 (+ 161us) server_negotiation.cc:407] Connection header received
0806 09:11:45.641734 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:45.641737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:45.641812 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:45.641907 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:45.643032 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.644257 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.645792 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:45.646308 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:45.646404 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:45.650808 (+ 4404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:37845: BlockingRecv error: recv got EOF from 127.3.69.2:37845 (error 108)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
Aug 06 09:11:46 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:46 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:46.075348 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:46.065905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:34777)
0806 09:11:46.066754 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:46.066770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:46.066930 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:46.067295 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:46.067300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:46.067751 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:46.067936 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:46.067945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.069000 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.069006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:46.070050 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:46.070062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.070832 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.070838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:46.071013 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:46.071637 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:46.071655 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:46.074855 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":246,"threads_started":1}
W20260806 09:11:46.075402 16455 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:46.065950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:34777 (local address 127.3.69.62:39371)
0806 09:11:46.066124 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:46.066129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:46.066860 (+ 731us) server_negotiation.cc:407] Connection header received
0806 09:11:46.067061 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:46.067065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:46.067146 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:46.067270 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:46.068084 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.068859 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.070186 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.070668 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.070801 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:46.075154 (+ 4353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:34777: BlockingRecv error: recv got EOF from 127.3.69.3:34777 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 09:11:46 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:46 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:46.128700 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:46.119592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:53571)
0806 09:11:46.120150 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:46.120166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:46.120281 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:46.120653 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:46.120657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:46.121204 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:46.121439 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:46.121449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.122610 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.122614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:46.123594 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:46.123607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.124394 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.124401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:46.124653 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:46.125349 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:46.125373 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:46.128436 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":319,"threads_started":1}
W20260806 09:11:46.128885 16455 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:46.119793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:53571 (local address 127.3.69.62:39371)
0806 09:11:46.119977 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:46.119982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:46.120242 (+ 260us) server_negotiation.cc:407] Connection header received
0806 09:11:46.120378 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:46.120381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:46.120445 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:46.120551 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:46.121645 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.122417 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.123752 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.124188 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.124300 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:46.128657 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:53571: BlockingRecv error: recv got EOF from 127.3.69.1:53571 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 09:11:46 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:46 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:46.660698 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:46.652052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:45419)
0806 09:11:46.652676 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:46.652704 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:46.652832 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:46.653115 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:46.653118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:46.653645 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:46.653804 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:46.653814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.654969 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.654973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:46.655968 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:46.655980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.656662 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.656667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:46.656784 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:46.657356 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:46.657372 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:46.660459 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":336,"threads_started":1}
W20260806 09:11:46.661013 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:46.652109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:45419 (local address 127.3.69.62:39371)
0806 09:11:46.652659 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:46.652664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:46.652785 (+ 121us) server_negotiation.cc:407] Connection header received
0806 09:11:46.652924 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:46.652928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:46.652989 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:46.653075 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:46.653923 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.654848 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.656080 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:46.656495 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:46.656584 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:46.660697 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:45419: BlockingRecv error: recv got EOF from 127.3.69.2:45419 (error 108)
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":323,"threads_started":1}
Aug 06 09:11:47 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:47 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:47.085049 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:47.076381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:36389)
0806 09:11:47.076851 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:47.076868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:47.076985 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:47.077268 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:47.077272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:47.077701 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:47.077937 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:47.077947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.078964 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.078968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:47.080004 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:47.080017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.080728 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.080733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:47.080849 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:47.081502 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:47.081520 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:47.084709 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
W20260806 09:11:47.085139 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:47.076426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:36389 (local address 127.3.69.62:39371)
0806 09:11:47.076680 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:47.076687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:47.076942 (+ 255us) server_negotiation.cc:407] Connection header received
0806 09:11:47.077081 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:47.077084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:47.077152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:47.077231 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:47.078071 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.078837 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.080149 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.080549 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.080653 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:47.084900 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:36389: BlockingRecv error: recv got EOF from 127.3.69.3:36389 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 09:11:47 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:47 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:47.139065 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:47.129833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:33487)
0806 09:11:47.130583 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:47.130603 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:47.130758 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:47.131327 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:47.131332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:47.131784 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:47.131951 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:47.131959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.133009 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.133013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:47.133987 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:47.133998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.134761 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.134767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:47.135002 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:47.135625 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:47.135642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:47.138699 (+ 3057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":446,"threads_started":1}
W20260806 09:11:47.139209 16462 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:47.129896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:33487 (local address 127.3.69.62:39371)
0806 09:11:47.130176 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:47.130185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:47.130691 (+ 506us) server_negotiation.cc:407] Connection header received
0806 09:11:47.130889 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:47.130895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:47.130983 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:47.131128 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:47.132078 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.132893 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.134133 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.134587 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.134733 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:47.138905 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:33487: BlockingRecv error: recv got EOF from 127.3.69.1:33487 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 09:11:47 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:47 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:47.674908 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:47.661722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:58743)
0806 09:11:47.662232 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:47.662249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:47.662356 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:47.662852 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:47.662855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:47.663460 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:47.663694 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:47.663709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.665243 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.665251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:47.666428 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:47.666441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.667652 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.667659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:47.667838 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:47.671458 (+ 3620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:47.671476 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:47.674627 (+ 3151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":251,"threads_started":1}
W20260806 09:11:47.675343 16469 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:47.661779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:58743 (local address 127.3.69.62:39371)
0806 09:11:47.662593 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:47.662597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:47.662612 (+ 15us) server_negotiation.cc:407] Connection header received
0806 09:11:47.662657 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:47.662660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:47.662723 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:47.662812 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:47.663852 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.665036 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.666605 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:47.667482 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:47.667612 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:47.674828 (+ 7216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:58743: BlockingRecv error: recv got EOF from 127.3.69.2:58743 (error 108)
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":606,"threads_started":1}
Aug 06 09:11:48 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:48 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:48.094681 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:48.086075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:43893)
0806 09:11:48.086640 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:48.086656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:48.086773 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:48.087110 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:48.087115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:48.087561 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:48.087743 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:48.087752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.088733 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.088737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:48.089673 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:48.089683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.090312 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.090315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:48.090430 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:48.091018 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:48.091035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:48.094402 (+ 3367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":300,"threads_started":1}
W20260806 09:11:48.094846 16469 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:48.086037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:43893 (local address 127.3.69.62:39371)
0806 09:11:48.086209 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:48.086214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:48.086725 (+ 511us) server_negotiation.cc:407] Connection header received
0806 09:11:48.086868 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:48.086872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:48.086936 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:48.087017 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:48.087865 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.088627 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.089788 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.090191 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.090279 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:48.094605 (+ 4326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:43893: BlockingRecv error: recv got EOF from 127.3.69.3:43893 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 09:11:48 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:48 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:48.149322 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:48.140253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:41527)
0806 09:11:48.140823 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:48.140846 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:48.141013 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:48.141447 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:48.141453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:48.141847 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:48.142088 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:48.142097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.143273 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.143277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:48.144346 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:48.144359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.145154 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.145159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:48.145290 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:48.146006 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:48.146024 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:48.149083 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":334,"threads_started":1}
W20260806 09:11:48.149539 16469 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:48.140322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:41527 (local address 127.3.69.62:39371)
0806 09:11:48.140533 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:48.140539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:48.140934 (+ 395us) server_negotiation.cc:407] Connection header received
0806 09:11:48.141169 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:48.141178 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:48.141258 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:48.141378 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:48.142215 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.143126 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.144486 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.144954 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.145060 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:48.149285 (+ 4225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:41527: BlockingRecv error: recv got EOF from 127.3.69.1:41527 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 09:11:48 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:48 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:48.684321 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:48.676050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:56111)
0806 09:11:48.676830 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:48.676847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:48.676954 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:48.677238 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:48.677248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:48.677608 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:48.677765 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:48.677773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.678772 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.678775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:48.679810 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:48.679822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.680502 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.680506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:48.680626 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:48.681199 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:48.681216 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:48.684050 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":567,"threads_started":1}
W20260806 09:11:48.684525 16473 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:48.676104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:56111 (local address 127.3.69.62:39371)
0806 09:11:48.676753 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:48.676759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:48.676919 (+ 160us) server_negotiation.cc:407] Connection header received
0806 09:11:48.677033 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:48.677037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:48.677097 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:48.677186 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:48.677905 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.678650 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.679943 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:48.680375 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:48.680467 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:48.684277 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:56111: BlockingRecv error: recv got EOF from 127.3.69.2:56111 (error 108)
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":350,"threads_started":1}
Aug 06 09:11:49 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:49 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:49.105144 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:49.095682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:51367)
0806 09:11:49.096258 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:49.096275 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:49.096422 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:49.096763 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:49.096768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:49.097277 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:49.097465 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:49.097475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.098589 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.098595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:49.099675 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:49.099688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.100409 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.100415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:49.100604 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:49.101260 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:49.101284 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:49.104792 (+ 3508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":338,"threads_started":1}
W20260806 09:11:49.105252 16473 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:49.095859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:51367 (local address 127.3.69.62:39371)
0806 09:11:49.096092 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:49.096118 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:49.096359 (+ 241us) server_negotiation.cc:407] Connection header received
0806 09:11:49.096553 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:49.096563 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:49.096627 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:49.096729 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:49.097624 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.098435 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.099818 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.100248 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.100379 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:49.105006 (+ 4627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:51367: BlockingRecv error: recv got EOF from 127.3.69.3:51367 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 09:11:49 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:49 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:49.159277 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:49.150235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:45407)
0806 09:11:49.150812 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:49.150829 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:49.150948 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:49.151378 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:49.151381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:49.152002 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:49.152272 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:49.152286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.153513 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.153517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:49.154524 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:49.154536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.155291 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.155295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:49.155408 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:49.156045 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:49.156064 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:49.158955 (+ 2891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":322,"threads_started":1}
W20260806 09:11:49.159487 16473 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:49.150363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:45407 (local address 127.3.69.62:39371)
0806 09:11:49.150565 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:49.150570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:49.150919 (+ 349us) server_negotiation.cc:407] Connection header received
0806 09:11:49.151068 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:49.151094 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:49.151202 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:49.151331 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:49.152440 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.153373 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.154656 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.155151 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.155265 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:49.159215 (+ 3950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:45407: BlockingRecv error: recv got EOF from 127.3.69.1:45407 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 09:11:49 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:49 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:49.694362 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:49.685240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:35987)
0806 09:11:49.685742 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:49.685764 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:49.685900 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:49.686527 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:49.686532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:49.687140 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:49.687402 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:49.687416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.688453 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.688458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:49.689501 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:49.689515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.690250 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.690255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:49.690375 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:49.691026 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:49.691044 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:49.694077 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":253,"threads_started":1}
W20260806 09:11:49.694602 16479 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:49.685360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:35987 (local address 127.3.69.62:39371)
0806 09:11:49.686155 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:49.686161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:49.686190 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:11:49.686253 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:49.686261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:49.686356 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:49.686462 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:49.687555 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.688313 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.689649 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:49.690086 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:49.690196 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:49.694344 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:35987: BlockingRecv error: recv got EOF from 127.3.69.2:35987 (error 108)
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":545,"threads_started":1}
Aug 06 09:11:50 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:50 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:50.114864 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:50.106123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:45025)
0806 09:11:50.106664 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:50.106682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:50.106858 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:50.107264 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:50.107272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:50.107750 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:50.107936 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:50.107944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.108933 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.108936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:50.109898 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:50.109909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.110626 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.110632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:50.110766 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:50.111414 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:50.111433 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:50.114533 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":314,"threads_started":1}
W20260806 09:11:50.115156 16479 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:50.106226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:45025 (local address 127.3.69.62:39371)
0806 09:11:50.106432 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:50.106440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:50.106789 (+ 349us) server_negotiation.cc:407] Connection header received
0806 09:11:50.106987 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:50.106991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:50.107058 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:50.107200 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:50.108063 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.108824 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.110013 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.110411 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.110513 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:50.114819 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:45025: BlockingRecv error: recv got EOF from 127.3.69.3:45025 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 09:11:50 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:50 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:50.169297 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:50.160158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:40589)
0806 09:11:50.160724 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:50.160741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:50.160907 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:50.161275 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:50.161280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:50.161820 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:50.161995 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:50.162004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.163232 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.163242 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:50.164370 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:50.164382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.165144 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.165151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:50.165406 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:50.166005 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:50.166023 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:50.169065 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
W20260806 09:11:50.169538 16479 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:50.160200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:40589 (local address 127.3.69.62:39371)
0806 09:11:50.160435 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:50.160443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:50.160836 (+ 393us) server_negotiation.cc:407] Connection header received
0806 09:11:50.161055 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:50.161059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:50.161136 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:50.161241 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:50.162150 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.163068 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.164540 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.164987 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.165159 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:50.169250 (+ 4091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:40589: BlockingRecv error: recv got EOF from 127.3.69.1:40589 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 09:11:50 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:50 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:50.704455 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:50.695301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:59027)
0806 09:11:50.696103 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:50.696120 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:50.696287 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:50.696717 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:50.696721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:50.697275 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:50.697495 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:50.697512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.698522 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.698526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:50.699580 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:50.699593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.700323 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.700327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:50.700454 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:50.701041 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:50.701058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:50.704113 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":287,"threads_started":1}
W20260806 09:11:50.704579 16483 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:50.695342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:59027 (local address 127.3.69.62:39371)
0806 09:11:50.695893 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:50.695897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:50.696220 (+ 323us) server_negotiation.cc:407] Connection header received
0806 09:11:50.696428 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:50.696431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:50.696536 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:50.696655 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:50.697629 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.698400 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.699717 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:50.700191 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:50.700290 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:50.704316 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:59027: BlockingRecv error: recv got EOF from 127.3.69.2:59027 (error 108)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
Aug 06 09:11:51 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:51 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:51.125320 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:51.116186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:50261)
0806 09:11:51.116966 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:51.116983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:51.117145 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:51.117573 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:51.117578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:51.117965 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:51.118128 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:51.118136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.119240 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.119244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:51.120454 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:51.120467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.121257 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.121261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:51.121443 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:51.122124 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:51.122141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:51.125083 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":507,"threads_started":1}
W20260806 09:11:51.125507 16483 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:51.116348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:50261 (local address 127.3.69.62:39371)
0806 09:11:51.116538 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:51.116543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:51.117078 (+ 535us) server_negotiation.cc:407] Connection header received
0806 09:11:51.117293 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:51.117298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:51.117386 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:51.117492 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:51.118292 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.119109 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.120612 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.121096 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.121234 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:51.125280 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:50261: BlockingRecv error: recv got EOF from 127.3.69.3:50261 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 09:11:51 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:51 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:51.179167 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:51.170224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:46309)
0806 09:11:51.170787 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:51.170805 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:51.170944 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:51.171304 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:51.171307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:51.171702 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:51.171887 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:51.171895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.172994 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.172998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:51.173927 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:51.173939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.174631 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.174636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:51.174761 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:51.175567 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:51.175591 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:51.178864 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":268,"threads_started":1}
W20260806 09:11:51.179365 16483 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:51.170337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:46309 (local address 127.3.69.62:39371)
0806 09:11:51.170583 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:51.170590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:51.170883 (+ 293us) server_negotiation.cc:407] Connection header received
0806 09:11:51.171058 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:51.171063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:51.171166 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:51.171258 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:51.172013 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.172842 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.174048 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.174454 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.174542 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:51.179062 (+ 4520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:46309: BlockingRecv error: recv got EOF from 127.3.69.1:46309 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 09:11:51 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:51 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:51.713678 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:51.705401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:60975)
0806 09:11:51.705959 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:51.705977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:51.706085 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:51.706575 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:51.706579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:51.706990 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:51.707220 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:51.707230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.708184 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.708187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:51.709107 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:51.709118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.709829 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.709834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:51.709951 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:51.710561 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:51.710578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:51.713435 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":268,"threads_started":1}
W20260806 09:11:51.713904 16491 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:51.705593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:60975 (local address 127.3.69.62:39371)
0806 09:11:51.706308 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:51.706312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:51.706328 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:11:51.706371 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:51.706375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:51.706433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:51.706523 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:51.707349 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.708062 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.709245 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:51.709669 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:51.709773 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:51.713642 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:60975: BlockingRecv error: recv got EOF from 127.3.69.2:60975 (error 108)
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":458,"threads_started":1}
Aug 06 09:11:52 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:52 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:52.134665 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:52.126320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:39253)
0806 09:11:52.126847 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:52.126865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:52.126984 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:52.127319 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:52.127323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:52.127756 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:52.127940 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:52.127949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.128957 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.128960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:52.129885 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:52.129896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.130694 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.130699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:52.130822 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:52.131466 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:52.131484 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:52.134426 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
W20260806 09:11:52.134853 16491 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:52.126413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:39253 (local address 127.3.69.62:39371)
0806 09:11:52.126670 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:52.126676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:52.126945 (+ 269us) server_negotiation.cc:407] Connection header received
0806 09:11:52.127115 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:52.127118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:52.127184 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:52.127272 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:52.128077 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.128821 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.130012 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.130506 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.130610 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:52.134633 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:39253: BlockingRecv error: recv got EOF from 127.3.69.3:39253 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 06 09:11:52 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:52 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:52.189960 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:52.180196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:51399)
0806 09:11:52.180766 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:52.180787 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:52.180962 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:52.181333 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:52.181337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:52.181810 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:52.181975 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:52.181983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.183118 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.183123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:52.184559 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:52.184576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.185408 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.185412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:52.185541 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:52.186235 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:52.186254 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:52.189549 (+ 3295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":312,"threads_started":1}
W20260806 09:11:52.190066 16491 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:52.180286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:51399 (local address 127.3.69.62:39371)
0806 09:11:52.180780 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:52.180788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:52.180887 (+ 99us) server_negotiation.cc:407] Connection header received
0806 09:11:52.181118 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:52.181122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:52.181183 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:52.181294 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:52.182134 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.182940 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.184751 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.185267 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.185371 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:52.189787 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:51399: BlockingRecv error: recv got EOF from 127.3.69.1:51399 (error 108)
Metrics: {"server-negotiator.queue_time_us":370}
Aug 06 09:11:52 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:52 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:52.727005 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:52.714672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:43829)
0806 09:11:52.715292 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:52.715315 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:52.715450 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:52.715868 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:52.715873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:52.716314 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:52.716472 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:52.716481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.717540 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.717546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:52.718471 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:52.718483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.719227 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.719232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:52.719418 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:52.723289 (+ 3871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:52.723307 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:52.726625 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":280,"threads_started":1}
W20260806 09:11:52.727172 16495 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:52.714771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:43829 (local address 127.3.69.62:39371)
0806 09:11:52.715508 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:52.715516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:52.715534 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:11:52.715628 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:52.715631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:52.715707 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:52.715830 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:52.716639 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.717394 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.718623 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:52.719045 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:52.719218 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:52.726881 (+ 7663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:43829: BlockingRecv error: recv got EOF from 127.3.69.2:43829 (error 108)
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":484,"threads_started":1}
Aug 06 09:11:53 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:53 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.3@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:53.144930 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:53.135765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.3:46363)
0806 09:11:53.136254 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:53.136271 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:53.136427 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:53.136778 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:53.136783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:53.137199 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:53.137358 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:53.137366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.138491 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.138499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:53.139486 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:53.139498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.140274 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.140279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:53.140499 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:53.141208 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:53.141226 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:53.144676 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":258,"threads_started":1}
W20260806 09:11:53.145120 16495 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:53.135835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:46363 (local address 127.3.69.62:39371)
0806 09:11:53.136074 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:53.136081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:53.136361 (+ 280us) server_negotiation.cc:407] Connection header received
0806 09:11:53.136552 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:53.136557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:53.136634 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:53.136740 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:53.137506 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.138340 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.139633 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.140115 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.140265 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:53.144875 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.3:46363: BlockingRecv error: recv got EOF from 127.3.69.3:46363 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 09:11:53 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:53 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.1@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:53.200271 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:53.191470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.1:47003)
0806 09:11:53.192035 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:53.192051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:53.192208 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:53.192560 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:53.192565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:53.193051 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:53.193226 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:53.193234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.194253 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.194259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:53.195212 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:53.195224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.195944 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.195950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:53.196122 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:53.196708 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:53.196725 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:53.199885 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":332,"threads_started":1}
W20260806 09:11:53.200378 16495 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:53.191487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:47003 (local address 127.3.69.62:39371)
0806 09:11:53.191681 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:53.191704 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:53.192141 (+ 437us) server_negotiation.cc:407] Connection header received
0806 09:11:53.192329 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:53.192334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:53.192426 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:53.192531 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:53.193374 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.194104 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.195357 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.195787 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.195920 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:53.200092 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.1:47003: BlockingRecv error: recv got EOF from 127.3.69.1:47003 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 09:11:53 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:11:53 dist-test-slave-w4v5 krb5kdc[14981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.69.2@KRBTEST.COM for oryx/127.3.69.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:11:53.737697 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:53.728420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39371 (local address 127.3.69.2:35277)
0806 09:11:53.729023 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:53.729041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:53.729184 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:53.729628 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:53.729633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:53.730169 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:53.730481 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:53.730504 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.731653 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.731658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:53.732742 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:53.732755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.733488 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.733493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:53.733633 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:53.734225 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:11:53.734243 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:53.737329 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.69.62:39371: Server oryx/127.3.69.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":296,"threads_started":1}
W20260806 09:11:53.737884 16499 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:53.728482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:35277 (local address 127.3.69.62:39371)
0806 09:11:53.729119 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:53.729127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:53.729148 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:11:53.729342 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:53.729350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:53.729453 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:53.729555 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:53.730627 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.731492 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.732880 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.733333 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.733427 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:53.737584 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.69.2:35277: BlockingRecv error: recv got EOF from 127.3.69.2:35277 (error 108)
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":412,"threads_started":1}
I20260806 09:11:53.943334 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 15816
I20260806 09:11:53.956576 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:53.944597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.3.69.1:39791)
0806 09:11:53.945271 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:53.945283 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:53.945322 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:11:53.945383 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:53.945389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:53.945616 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:53.945805 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:53.947631 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.948697 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.949924 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.950496 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.950595 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:53.953213 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:53.953221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:53.953234 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:53.953272 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:53.956098 (+ 2826us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.69.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:11:53.956356 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59004: Request ticket server kudu/127.3.69.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":311,"threads_started":1}
W20260806 09:11:53.956686 16502 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59004: Request ticket server kudu/127.3.69.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:11:53.956934 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 15816: Not authorized: Client connection negotiation failed: client connection to 127.3.69.1:39791: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.69.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:11:53.957114 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 15816
I20260806 09:11:53.984530 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 16043
I20260806 09:11:53.998188 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:53.985634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.3.69.2:35505)
0806 09:11:53.986394 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:53.986402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:53.986435 (+ 33us) server_negotiation.cc:407] Connection header received
0806 09:11:53.986507 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:53.986516 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:53.986801 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:53.987002 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:53.988417 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.989726 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.990981 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:53.991779 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:53.991885 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:53.994479 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:53.994487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:53.994500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:53.994542 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:53.997532 (+ 2990us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.69.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:11:53.997854 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50816: Request ticket server kudu/127.3.69.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":383,"threads_started":1}
W20260806 09:11:53.998189 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 16043: Not authorized: Client connection negotiation failed: client connection to 127.3.69.2:35505: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.69.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:11:53.998319 16504 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50816: Request ticket server kudu/127.3.69.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:11:53.998442 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 16043
I20260806 09:11:54.021920 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 16221
I20260806 09:11:54.034818 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:54.023220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.3.69.3:33273)
0806 09:11:54.024037 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:54.024045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:54.024076 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:11:54.024152 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:54.024158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:54.024424 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:54.024639 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:54.025908 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:54.026992 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:54.028185 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:54.028720 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:54.028818 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:54.031413 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:54.031421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:54.031435 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:11:54.031474 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:54.034235 (+ 2761us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.69.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:11:54.034575 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34034: Request ticket server kudu/127.3.69.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":439,"threads_started":1}
W20260806 09:11:54.034950 16505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34034: Request ticket server kudu/127.3.69.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:11:54.034967 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 16221: Not authorized: Client connection negotiation failed: client connection to 127.3.69.3:33273: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.69.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:11:54.035159 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 16221
I20260806 09:11:54.063179 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 15721
I20260806 09:11:54.064821 15809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:54.234226 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 15721
2026-08-06T09:11:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21716 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T09:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:54Z Disabled control of system clock
I20260806 09:11:54.332388 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42791
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 09:11:54.693156 16512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:54.693449 16512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:54.693533 16512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:54.700897 16512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:54.701020 16512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:54.701066 16512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:54.701125 16512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:54.701170 16512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:54.710451 16512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:42791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:54.711773 16512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:54.713546 16512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:54.725502 16519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:54.725622 16518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:54.725556 16522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:54.726260 16512 server_base.cc:1034] running on GCE node
I20260806 09:11:54.727039 16512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:54.728505 16512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:54.729729 16512 hybrid_clock.cc:648] HybridClock initialized: now 1786007514729680 us; error 46 us; skew 500 ppm
I20260806 09:11:54.730077 16512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:54.732462 16512 webserver.cc:460] Webserver started at http://127.3.69.62:41899/ using document root <none> and password file <none>
I20260806 09:11:54.733160 16512 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:54.733302 16512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:54.733636 16512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:54.736671 16512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff"
format_stamp: "Formatted at 2026-08-06 09:11:54 on dist-test-slave-w4v5"
server_key: "ed4bac12ce9b57ab80629f73d0759429"
server_key_iv: "9295b3b512ba802db7d3d5ac44de48d9"
server_key_version: "encryptionkey@0"
I20260806 09:11:54.737460 16512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff"
format_stamp: "Formatted at 2026-08-06 09:11:54 on dist-test-slave-w4v5"
server_key: "ed4bac12ce9b57ab80629f73d0759429"
server_key_iv: "9295b3b512ba802db7d3d5ac44de48d9"
server_key_version: "encryptionkey@0"
I20260806 09:11:54.742506 16512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:11:54.745942 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:54.747330 16512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:54.747514 16512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff"
format_stamp: "Formatted at 2026-08-06 09:11:54 on dist-test-slave-w4v5"
server_key: "ed4bac12ce9b57ab80629f73d0759429"
server_key_iv: "9295b3b512ba802db7d3d5ac44de48d9"
server_key_version: "encryptionkey@0"
I20260806 09:11:54.747723 16512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:54.775589 16512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:54.776453 16512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:54.776770 16512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:54.797281 16512 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:42791
I20260806 09:11:54.797303 16598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:42791 every 8 connection(s)
I20260806 09:11:54.798671 16512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:54.802838 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:54.805920 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 16512
I20260806 09:11:54.806130 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:54.806905 3348 external_mini_cluster.cc:1428] Setting key c7618638e4b17d81aa48b559fa5fbe03
I20260806 09:11:54.815969 16599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff: Bootstrap starting.
I20260806 09:11:54.819497 16599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:54.820742 16599 log.cc:824] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:54.821375 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:54.809191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.3.69.62:42791)
0806 09:11:54.810216 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:54.810227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:54.810270 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:11:54.812012 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:54.812036 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:54.812525 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:54.812945 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:54.813580 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:54.814903 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:54.816524 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:54.817236 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:54.817468 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:54.818783 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:54.818790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:54.818805 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:54.819429 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:54.819526 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:54.819547 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:54.819677 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:54.819996 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 09:11:54.820323 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":365,"threads_started":1}
I20260806 09:11:54.824059 16599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff: No bootstrap required, opened a new log
I20260806 09:11:54.831781 16599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42791 } }
I20260806 09:11:54.832160 16599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:54.832295 16599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b86e2196a44f1cb5bf2b5b7cdb9dff, State: Initialized, Role: FOLLOWER
I20260806 09:11:54.832852 16599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42791 } }
I20260806 09:11:54.833089 16599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:54.833231 16599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:54.833407 16599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:54.835813 16599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42791 } }
I20260806 09:11:54.836364 16599 leader_election.cc:302] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50b86e2196a44f1cb5bf2b5b7cdb9dff; no voters:
I20260806 09:11:54.836956 16599 leader_election.cc:288] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:54.837095 16606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:54.837806 16606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [term 1 LEADER]: Becoming Leader. State: Replica: 50b86e2196a44f1cb5bf2b5b7cdb9dff, State: Running, Role: LEADER
I20260806 09:11:54.838421 16606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42791 } }
I20260806 09:11:54.839044 16599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:54.841750 16609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50b86e2196a44f1cb5bf2b5b7cdb9dff. Latest consensus state: current_term: 1 leader_uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42791 } } }
I20260806 09:11:54.842216 16609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:54.842401 16607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b86e2196a44f1cb5bf2b5b7cdb9dff" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42791 } } }
I20260806 09:11:54.842769 16607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:54.844877 16616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:54.849746 16616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:54.858201 16616 catalog_manager.cc:1347] Generated new cluster ID: 28f128f295334af2bb48a9deef6c6442
I20260806 09:11:54.858373 16616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:54.868736 16616 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:54.870604 16616 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:54.894667 16616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50b86e2196a44f1cb5bf2b5b7cdb9dff: Generated new TSK 0
I20260806 09:11:54.895670 16616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:11:54.903250 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42791
--builtin_ntp_servers=127.3.69.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 09:11:55.270328 16632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:55.270646 16632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:55.270793 16632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:55.278524 16632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:55.278638 16632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:55.278808 16632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:55.288229 16632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:55.289654 16632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:55.291594 16632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:55.305761 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:55.305874 16637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:55.305786 16643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:55.306694 16632 server_base.cc:1034] running on GCE node
I20260806 09:11:55.307469 16632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:55.308561 16632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:55.309829 16632 hybrid_clock.cc:648] HybridClock initialized: now 1786007515309768 us; error 74 us; skew 500 ppm
I20260806 09:11:55.310209 16632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:55.312959 16632 webserver.cc:460] Webserver started at http://127.3.69.1:43275/ using document root <none> and password file <none>
I20260806 09:11:55.313787 16632 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:55.313936 16632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:55.314319 16632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:55.317564 16632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "52d7754e9b1047d2902590d9fa112924"
format_stamp: "Formatted at 2026-08-06 09:11:55 on dist-test-slave-w4v5"
server_key: "d7f5778c3867819f990ac1e93a745323"
server_key_iv: "0aeb2ac14206e416ee8ba970cb763abe"
server_key_version: "encryptionkey@0"
I20260806 09:11:55.318511 16632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "52d7754e9b1047d2902590d9fa112924"
format_stamp: "Formatted at 2026-08-06 09:11:55 on dist-test-slave-w4v5"
server_key: "d7f5778c3867819f990ac1e93a745323"
server_key_iv: "0aeb2ac14206e416ee8ba970cb763abe"
server_key_version: "encryptionkey@0"
I20260806 09:11:55.324674 16632 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:11:55.328747 16652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:55.330260 16632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:11:55.330435 16632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "52d7754e9b1047d2902590d9fa112924"
format_stamp: "Formatted at 2026-08-06 09:11:55 on dist-test-slave-w4v5"
server_key: "d7f5778c3867819f990ac1e93a745323"
server_key_iv: "0aeb2ac14206e416ee8ba970cb763abe"
server_key_version: "encryptionkey@0"
I20260806 09:11:55.330675 16632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:55.343951 16632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:55.344861 16632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:55.345148 16632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:55.346247 16632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:55.347846 16632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:55.347973 16632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:55.348102 16632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:55.348198 16632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:55.387627 16632 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:39907
I20260806 09:11:55.387687 16799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:39907 every 8 connection(s)
I20260806 09:11:55.389004 16632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:55.391516 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 16632
I20260806 09:11:55.391752 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:11:55.392498 3348 external_mini_cluster.cc:1428] Setting key fddf5da6124dabb5b320ebc3105e7909
I20260806 09:11:55.396003 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42791
--builtin_ntp_servers=127.3.69.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 09:11:55.406191 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:55.392570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:52481 (local address 127.3.69.62:42791)
0806 09:11:55.393076 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:55.393083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:55.395324 (+ 2241us) server_negotiation.cc:407] Connection header received
0806 09:11:55.398155 (+ 2831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:55.398160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:55.398234 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:55.398332 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:55.399827 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:55.400545 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:55.403067 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:55.403533 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:55.403670 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:55.405358 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:55.405363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:55.405366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:55.405385 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:55.405426 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:55.405437 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:55.405514 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:55.405821 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 09:11:55.405903 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":247,"threads_started":1}
I20260806 09:11:55.407698 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:55.393222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:42791 (local address 127.3.69.1:52481)
0806 09:11:55.395158 (+ 1936us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:55.395209 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:55.397844 (+ 2635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:55.398602 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:55.398616 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:55.398768 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:55.399625 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:55.399652 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:55.400736 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:55.400744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:55.402903 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:55.402921 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:55.403785 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:55.403795 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:55.404141 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:55.404943 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:55.404949 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:55.405023 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:55.405132 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:55.405544 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:55.405559 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:55.405569 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:11:55.405816 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 09:11:55.406255 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"thread_start_us":1066,"threads_started":1}
I20260806 09:11:55.409330 16800 heartbeater.cc:347] Connected to a master server at 127.3.69.62:42791
I20260806 09:11:55.409650 16800 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:55.410439 16800 heartbeater.cc:511] Master 127.3.69.62:42791 requested a full tablet report, sending...
I20260806 09:11:55.412731 16555 ts_manager.cc:194] Registered new tserver with Master: 52d7754e9b1047d2902590d9fa112924 (127.3.69.1:39907)
I20260806 09:11:55.414202 16555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.1:52481
W20260806 09:11:55.795156 16805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:55.795518 16805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:55.795689 16805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:55.808073 16805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:55.808211 16805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:55.808374 16805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:55.821926 16805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:55.823309 16805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:55.825202 16805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:55.840076 16818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:55.840072 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:55.840153 16816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:55.841583 16805 server_base.cc:1034] running on GCE node
I20260806 09:11:55.842415 16805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:55.843669 16805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:55.844933 16805 hybrid_clock.cc:648] HybridClock initialized: now 1786007515844789 us; error 89 us; skew 500 ppm
I20260806 09:11:55.845405 16805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:55.848141 16805 webserver.cc:460] Webserver started at http://127.3.69.2:38599/ using document root <none> and password file <none>
I20260806 09:11:55.849011 16805 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:55.849114 16805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:55.849486 16805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:55.853152 16805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "8352074214e44920b6ed87572aaccc38"
format_stamp: "Formatted at 2026-08-06 09:11:55 on dist-test-slave-w4v5"
server_key: "39ef537dc9c6ece65d004304bc215099"
server_key_iv: "b827db005dd58db0086c0df1f93c99a0"
server_key_version: "encryptionkey@0"
I20260806 09:11:55.854169 16805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "8352074214e44920b6ed87572aaccc38"
format_stamp: "Formatted at 2026-08-06 09:11:55 on dist-test-slave-w4v5"
server_key: "39ef537dc9c6ece65d004304bc215099"
server_key_iv: "b827db005dd58db0086c0df1f93c99a0"
server_key_version: "encryptionkey@0"
I20260806 09:11:55.859491 16805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:11:55.862957 16826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:55.864367 16805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:11:55.864553 16805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "8352074214e44920b6ed87572aaccc38"
format_stamp: "Formatted at 2026-08-06 09:11:55 on dist-test-slave-w4v5"
server_key: "39ef537dc9c6ece65d004304bc215099"
server_key_iv: "b827db005dd58db0086c0df1f93c99a0"
server_key_version: "encryptionkey@0"
I20260806 09:11:55.864780 16805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:55.881102 16805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:55.881958 16805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:55.882277 16805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:55.883405 16805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:55.885123 16805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:55.885248 16805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:55.885393 16805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:55.885484 16805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:55.936489 16805 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:34987
I20260806 09:11:55.936559 16985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:34987 every 8 connection(s)
I20260806 09:11:55.937831 16805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:55.946209 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 16805
I20260806 09:11:55.946434 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:11:55.947235 3348 external_mini_cluster.cc:1428] Setting key 13c57957e3ecc6cc772a692e960b7ab3
I20260806 09:11:55.950453 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42791
--builtin_ntp_servers=127.3.69.20:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 09:11:55.953123 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:55.940799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:39579 (local address 127.3.69.62:42791)
0806 09:11:55.941251 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:55.941256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:55.942579 (+ 1323us) server_negotiation.cc:407] Connection header received
0806 09:11:55.944474 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:55.944479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:55.944593 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:55.944761 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:55.946499 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:55.947764 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:55.949611 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:55.950279 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:55.950413 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:55.952115 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:55.952124 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:55.952132 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:55.952156 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:55.952210 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:55.952228 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:55.952347 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:55.952556 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 09:11:55.952661 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":214,"threads_started":1}
I20260806 09:11:55.953878 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:55.941410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:42791 (local address 127.3.69.2:39579)
0806 09:11:55.942423 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:55.942459 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:55.944194 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:55.944946 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:55.944953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:55.945078 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:55.945671 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:55.945688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:55.947988 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:55.947995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:55.949469 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:55.949481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:55.950507 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:55.950512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:55.950917 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:55.951702 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:55.951705 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:55.951789 (+ 84us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:55.951908 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:55.952352 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:55.952360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:55.952367 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 09:11:55.952513 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 09:11:55.952799 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":428,"threads_started":1}
I20260806 09:11:55.955312 16987 heartbeater.cc:347] Connected to a master server at 127.3.69.62:42791
I20260806 09:11:55.955627 16987 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:55.956463 16987 heartbeater.cc:511] Master 127.3.69.62:42791 requested a full tablet report, sending...
I20260806 09:11:55.957877 16555 ts_manager.cc:194] Registered new tserver with Master: 8352074214e44920b6ed87572aaccc38 (127.3.69.2:34987)
I20260806 09:11:55.958683 16555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.2:39579
W20260806 09:11:56.355433 16991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:56.355744 16991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:56.355834 16991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:56.363785 16991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:56.363886 16991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:56.363992 16991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:56.374339 16991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:42791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:56.375720 16991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:56.377461 16991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:56.392947 16997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:56.392971 16998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:56.394454 17003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:56.394449 16991 server_base.cc:1034] running on GCE node
I20260806 09:11:56.395284 16991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:56.396229 16991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:56.397440 16991 hybrid_clock.cc:648] HybridClock initialized: now 1786007516397404 us; error 39 us; skew 500 ppm
I20260806 09:11:56.397791 16991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:56.400434 16991 webserver.cc:460] Webserver started at http://127.3.69.3:33363/ using document root <none> and password file <none>
I20260806 09:11:56.401135 16991 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:56.401229 16991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:56.401607 16991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:56.404575 16991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "11cb65c6c3e94361bfedce7d3ac6e554"
format_stamp: "Formatted at 2026-08-06 09:11:56 on dist-test-slave-w4v5"
server_key: "cd836804952be9db9cb3c09fd8fb0597"
server_key_iv: "a7eadc45449ecd98ed4a6cbcca18f1f9"
server_key_version: "encryptionkey@0"
I20260806 09:11:56.405407 16991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "11cb65c6c3e94361bfedce7d3ac6e554"
format_stamp: "Formatted at 2026-08-06 09:11:56 on dist-test-slave-w4v5"
server_key: "cd836804952be9db9cb3c09fd8fb0597"
server_key_iv: "a7eadc45449ecd98ed4a6cbcca18f1f9"
server_key_version: "encryptionkey@0"
I20260806 09:11:56.411029 16991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:11:56.414983 17013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:56.416455 16991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:11:56.416736 16991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "11cb65c6c3e94361bfedce7d3ac6e554"
format_stamp: "Formatted at 2026-08-06 09:11:56 on dist-test-slave-w4v5"
server_key: "cd836804952be9db9cb3c09fd8fb0597"
server_key_iv: "a7eadc45449ecd98ed4a6cbcca18f1f9"
server_key_version: "encryptionkey@0"
I20260806 09:11:56.417047 16991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:56.417333 16800 heartbeater.cc:503] Master 127.3.69.62:42791 was elected leader, sending a full tablet report...
I20260806 09:11:56.432334 16991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:56.433580 16991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:56.433957 16991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:56.435155 16991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:56.436964 16991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:56.437090 16991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:56.437227 16991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:56.437326 16991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:56.483031 16991 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:43015
I20260806 09:11:56.483248 17158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:43015 every 8 connection(s)
I20260806 09:11:56.484579 16991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:56.492233 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 16991
I20260806 09:11:56.492455 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:11:56.492918 3348 external_mini_cluster.cc:1428] Setting key e7a9422ebf01c3f1b699eab5f2d12fbd
I20260806 09:11:56.498320 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:56.487285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:36359 (local address 127.3.69.62:42791)
0806 09:11:56.487738 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:56.487743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:56.488932 (+ 1189us) server_negotiation.cc:407] Connection header received
0806 09:11:56.490777 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:56.490780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:56.490857 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:56.490956 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:56.492220 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.493086 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.495183 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.495631 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.495781 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:56.497375 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:56.497380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:56.497383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:56.497401 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:56.497441 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:56.497452 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:56.497552 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:56.497834 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 09:11:56.497935 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260806 09:11:56.499428 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:56.487853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:42791 (local address 127.3.69.3:36359)
0806 09:11:56.488776 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:56.488812 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:56.490518 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:56.491218 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:56.491226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:56.491308 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:56.492042 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:56.492070 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.493247 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.493255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:56.494976 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:11:56.494989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.495870 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.495877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:11:56.496257 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:56.497042 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:56.497045 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:56.497088 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:56.497159 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:56.497551 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:56.497573 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:56.497583 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:11:56.497820 (+ 237us) client_negotiation.cc:240] Negotiation successful
0806 09:11:56.498201 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":289,"threads_started":1}
I20260806 09:11:56.500790 17159 heartbeater.cc:347] Connected to a master server at 127.3.69.62:42791
I20260806 09:11:56.501111 17159 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:56.502317 17159 heartbeater.cc:511] Master 127.3.69.62:42791 requested a full tablet report, sending...
I20260806 09:11:56.503985 16555 ts_manager.cc:194] Registered new tserver with Master: 11cb65c6c3e94361bfedce7d3ac6e554 (127.3.69.3:43015)
I20260806 09:11:56.504801 16555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.3:36359
I20260806 09:11:56.508564 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 09:11:56.515118 17162 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:56.513358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.3.69.62:42791)
0806 09:11:56.513614 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:56.513621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:56.514023 (+ 402us) server_negotiation.cc:407] Connection header received
0806 09:11:56.514179 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:56.514182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:56.514242 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:56.514358 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:56.514859 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54614: BlockingRecv error: recv got EOF from 127.0.0.1:54614 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:11:56.515710 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 16632
I20260806 09:11:56.523658 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:56.516720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.3.69.1:39907)
0806 09:11:56.517281 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:56.517288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:56.517314 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:11:56.517369 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:56.517375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:56.517571 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:56.517705 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:56.518053 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.519127 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.520393 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.520949 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.521102 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:56.522164 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:56.522175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:56.522195 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:56.522824 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:56.522869 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:56.522889 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:56.522978 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:56.523165 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 09:11:56.523296 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260806 09:11:56.525096 16791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:56.606775 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 16632
I20260806 09:11:56.629036 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 16805
I20260806 09:11:56.637641 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:56.630133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.3.69.2:34987)
0806 09:11:56.630728 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:56.630743 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:56.630770 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:11:56.630839 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:56.630845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:56.631065 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:56.631261 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:56.631700 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.632931 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.634349 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.634934 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.635042 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:56.636114 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:56.636122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:56.636136 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:56.636766 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:56.636821 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:56.636841 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:56.636933 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:56.637069 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 09:11:56.637237 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":319,"threads_started":1}
I20260806 09:11:56.638980 16979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:56.722752 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 16805
I20260806 09:11:56.748198 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 16991
I20260806 09:11:56.756973 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:56.749340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.3.69.3:43015)
0806 09:11:56.750036 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:56.750043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:56.750074 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:11:56.750141 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:56.750148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:56.750362 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:56.750511 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:56.750940 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.752178 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.753425 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:11:56.754016 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:11:56.754126 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:11:56.755389 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:56.755398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:56.755411 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:56.755993 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:56.756038 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:56.756062 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:56.756164 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:56.756329 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 09:11:56.756481 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":383,"threads_started":1}
I20260806 09:11:56.758665 17152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:56.846477 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 16991
I20260806 09:11:56.870968 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 16512
I20260806 09:11:56.872340 16593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:56.975669 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 16512
2026-08-06T09:11:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2734 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T09:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:11:57Z Disabled control of system clock
I20260806 09:11:57.072630 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45139
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:45139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 09:11:57.410454 17185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:57.410837 17185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:57.410991 17185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:57.418172 17185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:11:57.418322 17185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:57.418416 17185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:57.418538 17185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:11:57.418617 17185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:11:57.427843 17185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:45139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:57.429179 17185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:57.431052 17185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:57.443233 17194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:57.443548 17191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:57.443497 17192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:57.444435 17185 server_base.cc:1034] running on GCE node
I20260806 09:11:57.445338 17185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:57.446805 17185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:57.448057 17185 hybrid_clock.cc:648] HybridClock initialized: now 1786007517447985 us; error 61 us; skew 500 ppm
I20260806 09:11:57.448455 17185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:57.450877 17185 webserver.cc:460] Webserver started at http://127.3.69.62:44165/ using document root <none> and password file <none>
I20260806 09:11:57.451663 17185 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:57.451798 17185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:57.452147 17185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:57.455236 17185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "1c79f1c794af4923a95bd669f78c935b"
format_stamp: "Formatted at 2026-08-06 09:11:57 on dist-test-slave-w4v5"
server_key: "79b71c8b94e96f0e4d2d97da83fed3e7"
server_key_iv: "3a477da4ca47177f3c90a5ba268b7b28"
server_key_version: "encryptionkey@0"
I20260806 09:11:57.456187 17185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "1c79f1c794af4923a95bd669f78c935b"
format_stamp: "Formatted at 2026-08-06 09:11:57 on dist-test-slave-w4v5"
server_key: "79b71c8b94e96f0e4d2d97da83fed3e7"
server_key_iv: "3a477da4ca47177f3c90a5ba268b7b28"
server_key_version: "encryptionkey@0"
I20260806 09:11:57.461567 17185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:11:57.465160 17202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:57.466569 17185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:11:57.466779 17185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "1c79f1c794af4923a95bd669f78c935b"
format_stamp: "Formatted at 2026-08-06 09:11:57 on dist-test-slave-w4v5"
server_key: "79b71c8b94e96f0e4d2d97da83fed3e7"
server_key_iv: "3a477da4ca47177f3c90a5ba268b7b28"
server_key_version: "encryptionkey@0"
I20260806 09:11:57.467027 17185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:57.473054 17185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:57.473843 17185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:57.474157 17185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:57.493913 17185 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:45139
I20260806 09:11:57.493966 17273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:45139 every 8 connection(s)
I20260806 09:11:57.495339 17185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:11:57.498931 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 17185
I20260806 09:11:57.499207 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:11:57.499408 17275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:11:57.499961 3348 external_mini_cluster.cc:1428] Setting key 539d36a1bec345246707bdf0a9d4f9cd
I20260806 09:11:57.508805 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:57.502001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.3.69.62:45139)
0806 09:11:57.503167 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:57.503178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:57.503222 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:11:57.505255 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:57.505277 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:57.505680 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:57.506012 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:57.507163 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:57.507169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:57.507181 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:57.507196 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:57.507262 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:57.507278 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:57.507441 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:57.507673 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 09:11:57.507942 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":440,"threads_started":1}
I20260806 09:11:57.513832 17275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b: Bootstrap starting.
I20260806 09:11:57.516844 17275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b: Neither blocks nor log segments found. Creating new log.
I20260806 09:11:57.517949 17275 log.cc:824] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:11:57.520884 17275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b: No bootstrap required, opened a new log
I20260806 09:11:57.528323 17275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c79f1c794af4923a95bd669f78c935b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45139 } }
I20260806 09:11:57.528673 17275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:11:57.528750 17275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c79f1c794af4923a95bd669f78c935b, State: Initialized, Role: FOLLOWER
I20260806 09:11:57.529240 17275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c79f1c794af4923a95bd669f78c935b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45139 } }
I20260806 09:11:57.529435 17275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:11:57.529541 17275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:11:57.529660 17275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:11:57.531860 17275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c79f1c794af4923a95bd669f78c935b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45139 } }
I20260806 09:11:57.532294 17275 leader_election.cc:302] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c79f1c794af4923a95bd669f78c935b; no voters:
I20260806 09:11:57.532847 17275 leader_election.cc:288] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:11:57.532963 17281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:11:57.533735 17281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [term 1 LEADER]: Becoming Leader. State: Replica: 1c79f1c794af4923a95bd669f78c935b, State: Running, Role: LEADER
I20260806 09:11:57.534268 17281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c79f1c794af4923a95bd669f78c935b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45139 } }
I20260806 09:11:57.535182 17275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:11:57.537333 17285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c79f1c794af4923a95bd669f78c935b. Latest consensus state: current_term: 1 leader_uuid: "1c79f1c794af4923a95bd669f78c935b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c79f1c794af4923a95bd669f78c935b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45139 } } }
I20260806 09:11:57.537245 17283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c79f1c794af4923a95bd669f78c935b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c79f1c794af4923a95bd669f78c935b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45139 } } }
I20260806 09:11:57.537747 17285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:57.537813 17283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b [sys.catalog]: This master's current role is: LEADER
I20260806 09:11:57.539345 17289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:11:57.544612 17289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:11:57.553527 17289 catalog_manager.cc:1347] Generated new cluster ID: 2efb3041f547427983fde35a7308fc91
I20260806 09:11:57.553632 17289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:11:57.594131 17289 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:11:57.594594 17289 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:11:57.601775 17289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c79f1c794af4923a95bd669f78c935b: Generated new TSK 0
I20260806 09:11:57.602589 17289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:11:57.621359 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45139
--builtin_ntp_servers=127.3.69.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 09:11:57.985791 17304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:57.986065 17304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:57.986163 17304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:57.993235 17304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:57.993336 17304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:57.993445 17304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:11:58.002200 17304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:58.003500 17304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:58.005296 17304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:58.019353 17310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:58.019316 17312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:58.019544 17316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:58.020190 17304 server_base.cc:1034] running on GCE node
I20260806 09:11:58.020854 17304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:58.021806 17304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:58.023043 17304 hybrid_clock.cc:648] HybridClock initialized: now 1786007518022995 us; error 34 us; skew 500 ppm
I20260806 09:11:58.023445 17304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:58.030602 17304 webserver.cc:460] Webserver started at http://127.3.69.1:45085/ using document root <none> and password file <none>
I20260806 09:11:58.031447 17304 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:58.031589 17304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:58.031939 17304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:58.034849 17304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "f41696de3a9e43248abad691a7424e85"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "82bdcfb26dd9e31cd40c86017efbac00"
server_key_iv: "5265d5dc12d552cfe2d523cc16e6e507"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.035833 17304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "f41696de3a9e43248abad691a7424e85"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "82bdcfb26dd9e31cd40c86017efbac00"
server_key_iv: "5265d5dc12d552cfe2d523cc16e6e507"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.041492 17304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:11:58.044948 17325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.046232 17304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:11:58.046416 17304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "f41696de3a9e43248abad691a7424e85"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "82bdcfb26dd9e31cd40c86017efbac00"
server_key_iv: "5265d5dc12d552cfe2d523cc16e6e507"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.046646 17304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:58.052366 17304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:58.053133 17304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:58.053401 17304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:58.054307 17304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:58.055745 17304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:58.055871 17304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.055996 17304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:58.056083 17304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.091686 17304 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:34791
I20260806 09:11:58.091738 17478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:34791 every 8 connection(s)
I20260806 09:11:58.093086 17304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:11:58.094660 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 17304
I20260806 09:11:58.094894 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:11:58.095875 3348 external_mini_cluster.cc:1428] Setting key a897e59847f3c936fe26ac2b54d1862a
I20260806 09:11:58.098853 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45139
--builtin_ntp_servers=127.3.69.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 09:11:58.102046 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:58.095916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:45899 (local address 127.3.69.62:45139)
0806 09:11:58.096840 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:58.096848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:58.097877 (+ 1029us) server_negotiation.cc:407] Connection header received
0806 09:11:58.099655 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:58.099660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:58.099730 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:58.099865 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:58.101238 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:58.101243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:58.101248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:58.101266 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:58.101317 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:58.101330 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:58.101461 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:58.101657 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 09:11:58.101731 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":651,"threads_started":1}
I20260806 09:11:58.102937 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:58.096583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45139 (local address 127.3.69.1:45899)
0806 09:11:58.097711 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:58.097755 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:58.099380 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:58.100117 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:58.100125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:58.100217 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:58.100934 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:58.100936 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:58.100974 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:58.101058 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:58.101427 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:58.101439 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:58.101449 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:11:58.101750 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 09:11:58.102005 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":358,"threads_started":1}
I20260806 09:11:58.104493 17480 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45139
I20260806 09:11:58.104826 17480 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:58.105557 17480 heartbeater.cc:511] Master 127.3.69.62:45139 requested a full tablet report, sending...
I20260806 09:11:58.108043 17225 ts_manager.cc:194] Registered new tserver with Master: f41696de3a9e43248abad691a7424e85 (127.3.69.1:34791)
W20260806 09:11:58.437558 17485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:58.437839 17485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:58.437994 17485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:58.445531 17485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:58.445633 17485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:58.445832 17485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:11:58.455240 17485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:58.456586 17485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:58.458431 17485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:58.475648 17494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:58.476480 17495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:58.476521 17497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:58.477260 17485 server_base.cc:1034] running on GCE node
I20260806 09:11:58.477865 17485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:58.478820 17485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:58.480062 17485 hybrid_clock.cc:648] HybridClock initialized: now 1786007518479995 us; error 74 us; skew 500 ppm
I20260806 09:11:58.480406 17485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:58.482743 17485 webserver.cc:460] Webserver started at http://127.3.69.2:33255/ using document root <none> and password file <none>
I20260806 09:11:58.483489 17485 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:58.483620 17485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:58.483961 17485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:58.486801 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "4e1678067a9c423ba1de69a295502e49"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "9e1a8cf1946dc2e11f0de524c86dcff3"
server_key_iv: "ec2f3b0d41bcb56523aafa71379d8635"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.487648 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "4e1678067a9c423ba1de69a295502e49"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "9e1a8cf1946dc2e11f0de524c86dcff3"
server_key_iv: "ec2f3b0d41bcb56523aafa71379d8635"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.492949 17485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:11:58.496511 17505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.497891 17485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:11:58.498049 17485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "4e1678067a9c423ba1de69a295502e49"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "9e1a8cf1946dc2e11f0de524c86dcff3"
server_key_iv: "ec2f3b0d41bcb56523aafa71379d8635"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.498268 17485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:58.503980 17485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:58.504729 17485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:58.504997 17485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:58.506007 17485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:58.507452 17485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:58.507565 17485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.507694 17485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:58.507779 17485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.542511 17485 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:41235
I20260806 09:11:58.542572 17653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:41235 every 8 connection(s)
I20260806 09:11:58.543929 17485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:11:58.548606 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 17485
I20260806 09:11:58.548832 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:11:58.549310 3348 external_mini_cluster.cc:1428] Setting key b430a6dbbe47e8cb3527cf0ee247e5d9
I20260806 09:11:58.552417 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45139
--builtin_ntp_servers=127.3.69.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 09:11:58.553974 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:58.546634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:47531 (local address 127.3.69.62:45139)
0806 09:11:58.546818 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:58.546822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:58.548253 (+ 1431us) server_negotiation.cc:407] Connection header received
0806 09:11:58.550208 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:58.550213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:58.550298 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:58.550411 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:58.553198 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:58.553202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:58.553205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:58.553221 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:58.553254 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:58.553265 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:58.553397 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:58.553629 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 09:11:58.553702 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:11:58.554920 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:58.547125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45139 (local address 127.3.69.2:47531)
0806 09:11:58.548101 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:58.548138 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:58.549947 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:58.550728 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:58.550739 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:58.550853 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:58.552869 (+ 2016us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:58.552872 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:58.552911 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:58.552996 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:58.553372 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:58.553383 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:58.553391 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 09:11:58.553712 (+ 321us) client_negotiation.cc:240] Negotiation successful
0806 09:11:58.553974 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":322,"threads_started":1}
I20260806 09:11:58.556330 17655 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45139
I20260806 09:11:58.556622 17655 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:58.557197 17655 heartbeater.cc:511] Master 127.3.69.62:45139 requested a full tablet report, sending...
I20260806 09:11:58.558684 17225 ts_manager.cc:194] Registered new tserver with Master: 4e1678067a9c423ba1de69a295502e49 (127.3.69.2:41235)
W20260806 09:11:58.891737 17660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:11:58.892057 17660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:11:58.892216 17660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:11:58.899520 17660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:11:58.899643 17660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:11:58.899820 17660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:11:58.909193 17660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:11:58.910552 17660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:11:58.912441 17660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:11:58.925400 17667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:58.925447 17668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:11:58.925710 17670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:11:58.926086 17660 server_base.cc:1034] running on GCE node
I20260806 09:11:58.926699 17660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:11:58.927639 17660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:11:58.928860 17660 hybrid_clock.cc:648] HybridClock initialized: now 1786007518928816 us; error 61 us; skew 500 ppm
I20260806 09:11:58.929128 17660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:11:58.931361 17660 webserver.cc:460] Webserver started at http://127.3.69.3:38377/ using document root <none> and password file <none>
I20260806 09:11:58.931985 17660 fs_manager.cc:359] Metadata directory not provided
I20260806 09:11:58.932071 17660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:11:58.932345 17660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:11:58.935174 17660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "25467c5b83a044b8914fe2b1d34acbb0"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "43cd78ca7a70f97da885594f251c538c"
server_key_iv: "02cb78d6875fd3e31e101d568720b9e9"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.936004 17660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "25467c5b83a044b8914fe2b1d34acbb0"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "43cd78ca7a70f97da885594f251c538c"
server_key_iv: "02cb78d6875fd3e31e101d568720b9e9"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.941210 17660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 09:11:58.944612 17683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.945868 17660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:11:58.946048 17660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "25467c5b83a044b8914fe2b1d34acbb0"
format_stamp: "Formatted at 2026-08-06 09:11:58 on dist-test-slave-w4v5"
server_key: "43cd78ca7a70f97da885594f251c538c"
server_key_iv: "02cb78d6875fd3e31e101d568720b9e9"
server_key_version: "encryptionkey@0"
I20260806 09:11:58.946260 17660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:11:58.952126 17660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:11:58.952893 17660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:11:58.953166 17660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:11:58.954084 17660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:11:58.955528 17660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:11:58.955643 17660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.955783 17660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:11:58.955878 17660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:11:58.989145 17660 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:46853
I20260806 09:11:58.989259 17843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:46853 every 8 connection(s)
I20260806 09:11:58.990528 17660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:11:58.998688 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:58.993136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:58427 (local address 127.3.69.62:45139)
0806 09:11:58.993316 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:58.993321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:58.994653 (+ 1332us) server_negotiation.cc:407] Connection header received
0806 09:11:58.996467 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:58.996473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:58.996555 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:58.996675 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:58.997909 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:58.997915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:58.997920 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:58.997947 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:58.997988 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:58.998000 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:58.998127 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:58.998319 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 09:11:58.998393 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:11:58.999737 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:58.993555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45139 (local address 127.3.69.3:58427)
0806 09:11:58.994470 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 09:11:58.994522 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:11:58.996192 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:11:58.996898 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:11:58.996906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:11:58.997002 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:11:58.997617 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:11:58.997620 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:11:58.997657 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:11:58.997736 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:11:58.998098 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:11:58.998108 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:11:58.998115 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 09:11:58.998399 (+ 284us) client_negotiation.cc:240] Negotiation successful
0806 09:11:58.998681 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":301,"threads_started":1}
I20260806 09:11:59.000325 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 17660
I20260806 09:11:59.000541 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:11:59.001147 3348 external_mini_cluster.cc:1428] Setting key 69e752e0505ad35782af73650f3679a6
I20260806 09:11:59.001186 17844 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45139
I20260806 09:11:59.001469 17844 heartbeater.cc:464] Registering TS with master...
I20260806 09:11:59.002161 17844 heartbeater.cc:511] Master 127.3.69.62:45139 requested a full tablet report, sending...
I20260806 09:11:59.003765 17224 ts_manager.cc:194] Registered new tserver with Master: 25467c5b83a044b8914fe2b1d34acbb0 (127.3.69.3:46853)
I20260806 09:11:59.016845 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 09:11:59.023787 17483 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:11:59.021895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.3.69.62:45139)
0806 09:11:59.022106 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:59.022111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:59.022925 (+ 814us) server_negotiation.cc:407] Connection header received
0806 09:11:59.023054 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:59.023057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:59.023179 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:59.023272 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:59.023587 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37754: BlockingRecv error: recv got EOF from 127.0.0.1:37754 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 09:11:59.024531 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 17304
I20260806 09:11:59.028339 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:59.025575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.3.69.1:34791)
0806 09:11:59.026122 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:59.026129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:59.026155 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:11:59.026220 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:59.026225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:59.026426 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:59.026556 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:59.027631 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:59.027638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:59.027649 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:59.027680 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:59.027721 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:59.027734 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:59.027838 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:59.027923 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 09:11:59.028035 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":254,"threads_started":1}
I20260806 09:11:59.029544 17472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:59.110702 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 17304
I20260806 09:11:59.134656 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 17485
I20260806 09:11:59.138584 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:59.135723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.3.69.2:41235)
0806 09:11:59.136370 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:59.136378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:59.136405 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:11:59.136477 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:59.136484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:59.136705 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:59.136835 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:59.137759 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:59.137770 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:59.137788 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:59.137809 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:59.137870 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:59.137893 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:59.138038 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:59.138114 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 09:11:59.138228 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":304,"threads_started":1}
I20260806 09:11:59.139997 17647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:59.220824 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 17485
I20260806 09:11:59.246721 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 17660
I20260806 09:11:59.251009 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:11:59.247909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.3.69.3:46853)
0806 09:11:59.248560 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0806 09:11:59.248567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:11:59.248598 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:11:59.248660 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:11:59.248667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:11:59.248888 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:11:59.249059 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:11:59.250230 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:11:59.250239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:11:59.250251 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:11:59.250266 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:11:59.250312 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:11:59.250327 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:11:59.250457 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:11:59.250650 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 09:11:59.250733 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":332,"threads_started":1}
I20260806 09:11:59.252367 17836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:59.334111 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 17660
I20260806 09:11:59.357029 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 17185
I20260806 09:11:59.358487 17268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:11:59.460876 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 17185
2026-08-06T09:11:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2483 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:11:59 dist-test-slave-w4v5 krb5kdc[17867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:11:59 dist-test-slave-w4v5 krb5kdc[17867](info): set up 2 sockets
Aug 06 09:11:59 dist-test-slave-w4v5 krb5kdc[17867](info): commencing operation
krb5kdc: starting...
W20260806 09:12:00.777222 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.007s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:12:00 dist-test-slave-w4v5 krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:12:01.058777 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40035
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:12:01.400458 17887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:12:01.400785 17887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:12:01.400949 17887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:12:01.408716 17887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:12:01.408862 17887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:12:01.408955 17887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:12:01.409082 17887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:12:01.409157 17887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:12:01.420023 17887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:40035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:12:01.421504 17887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:12:01.422940 17887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:12:01.435714 17893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:01.435652 17896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:01.435675 17894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:12:01.436856 17887 server_base.cc:1034] running on GCE node
I20260806 09:12:01.437736 17887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:12:01.439289 17887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:12:01.440501 17887 hybrid_clock.cc:648] HybridClock initialized: now 1786007521440467 us; error 35 us; skew 500 ppm
I20260806 09:12:01.440878 17887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:12:01 dist-test-slave-w4v5 krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:01.444252 17887 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:12:01.446031 17887 webserver.cc:460] Webserver started at http://127.3.69.62:39865/ using document root <none> and password file <none>
I20260806 09:12:01.446784 17887 fs_manager.cc:359] Metadata directory not provided
I20260806 09:12:01.446923 17887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:12:01.447321 17887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:12:01.450441 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "1a16914c3b474cdaa05ba1dc3bd6d531"
format_stamp: "Formatted at 2026-08-06 09:12:01 on dist-test-slave-w4v5"
server_key: "57e9c089ede7125781403b48072ed233"
server_key_iv: "a1ff1e2846c813355894af3a4bf6c822"
server_key_version: "encryptionkey@0"
I20260806 09:12:01.451336 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "1a16914c3b474cdaa05ba1dc3bd6d531"
format_stamp: "Formatted at 2026-08-06 09:12:01 on dist-test-slave-w4v5"
server_key: "57e9c089ede7125781403b48072ed233"
server_key_iv: "a1ff1e2846c813355894af3a4bf6c822"
server_key_version: "encryptionkey@0"
I20260806 09:12:01.456612 17887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:12:01.460145 17908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:01.461531 17887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:12:01.461750 17887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "1a16914c3b474cdaa05ba1dc3bd6d531"
format_stamp: "Formatted at 2026-08-06 09:12:01 on dist-test-slave-w4v5"
server_key: "57e9c089ede7125781403b48072ed233"
server_key_iv: "a1ff1e2846c813355894af3a4bf6c822"
server_key_version: "encryptionkey@0"
I20260806 09:12:01.462014 17887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:12:01.485513 17887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:12:01.489976 17887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:12:01.490298 17887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:12:01.510699 17887 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:40035
I20260806 09:12:01.510749 17971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:40035 every 8 connection(s)
I20260806 09:12:01.512243 17887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:12:01.516129 17972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:12:01.516940 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 17887
I20260806 09:12:01.517198 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:12:01.518412 3348 external_mini_cluster.cc:1428] Setting key 7dc3eaa3c7cd387dab6a11622d04f819
I20260806 09:12:01.532835 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531: Bootstrap starting.
Aug 06 09:12:01 dist-test-slave-w4v5 krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:12:01.536938 17972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531: Neither blocks nor log segments found. Creating new log.
I20260806 09:12:01.538640 17972 log.cc:824] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531: Log is configured to *not* fsync() on all Append() calls
I20260806 09:12:01.542369 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531: No bootstrap required, opened a new log
I20260806 09:12:01.545584 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:01.520777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.3.69.62:40035)
0806 09:12:01.521615 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:01.521631 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:01.521675 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:12:01.523502 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:01.523526 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:01.523930 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:01.524292 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:01.526460 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:01.527858 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:01.529921 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:01.530648 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:01.530788 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:01.536248 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:01.536257 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:01.536272 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:01.536312 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:01.540470 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:01.541190 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:01.541199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:01.541209 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:01.541457 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:01.541840 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:01.541845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:01.541848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:01.542628 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:01.542998 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:01.543530 (+ 532us) server_negotiation.cc:299] Negotiation successful
0806 09:12:01.544017 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":402,"threads_started":1}
I20260806 09:12:01.551957 17972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a16914c3b474cdaa05ba1dc3bd6d531" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40035 } }
I20260806 09:12:01.552358 17972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:12:01.552500 17972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a16914c3b474cdaa05ba1dc3bd6d531, State: Initialized, Role: FOLLOWER
I20260806 09:12:01.553211 17972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a16914c3b474cdaa05ba1dc3bd6d531" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40035 } }
I20260806 09:12:01.553437 17972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:12:01.553545 17972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:12:01.553674 17972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:12:01.556157 17972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a16914c3b474cdaa05ba1dc3bd6d531" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40035 } }
I20260806 09:12:01.556643 17972 leader_election.cc:302] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a16914c3b474cdaa05ba1dc3bd6d531; no voters:
I20260806 09:12:01.557204 17972 leader_election.cc:288] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:12:01.557361 17979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:12:01.558161 17979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [term 1 LEADER]: Becoming Leader. State: Replica: 1a16914c3b474cdaa05ba1dc3bd6d531, State: Running, Role: LEADER
I20260806 09:12:01.558770 17979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a16914c3b474cdaa05ba1dc3bd6d531" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40035 } }
I20260806 09:12:01.559599 17972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:12:01.561817 17981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a16914c3b474cdaa05ba1dc3bd6d531. Latest consensus state: current_term: 1 leader_uuid: "1a16914c3b474cdaa05ba1dc3bd6d531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a16914c3b474cdaa05ba1dc3bd6d531" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40035 } } }
I20260806 09:12:01.562179 17981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [sys.catalog]: This master's current role is: LEADER
I20260806 09:12:01.563656 17980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a16914c3b474cdaa05ba1dc3bd6d531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a16914c3b474cdaa05ba1dc3bd6d531" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40035 } } }
I20260806 09:12:01.564131 17980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531 [sys.catalog]: This master's current role is: LEADER
I20260806 09:12:01.567034 17988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:12:01.571823 17988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:12:01.581123 17988 catalog_manager.cc:1347] Generated new cluster ID: ce4754f68ea64cb6a332abeafb819abf
I20260806 09:12:01.581269 17988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:12:01.602710 17988 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:12:01.604790 17988 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:12:01.629803 17988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531: Generated new TSK 0
I20260806 09:12:01.630712 17988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:12:01.710582 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40035
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:12:02.076059 18010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:12:02.076422 18010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:12:02.076594 18010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:12:02.086874 18010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:12:02.087049 18010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:12:02.087265 18010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:12:02.097242 18010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:12:02.098598 18010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:12:02.099990 18010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:12:02.113178 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:02.113271 18019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:02.113279 18022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:12:02.114392 18010 server_base.cc:1034] running on GCE node
I20260806 09:12:02.115041 18010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:12:02.117133 18010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:12:02.118371 18010 hybrid_clock.cc:648] HybridClock initialized: now 1786007522118326 us; error 54 us; skew 500 ppm
I20260806 09:12:02.118738 18010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:12:02 dist-test-slave-w4v5 krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:02.122056 18010 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:12:02.123562 18010 webserver.cc:460] Webserver started at http://127.3.69.1:42567/ using document root <none> and password file <none>
I20260806 09:12:02.124253 18010 fs_manager.cc:359] Metadata directory not provided
I20260806 09:12:02.124405 18010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:12:02.124750 18010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:12:02.127779 18010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "5dce315adf2748b09db555e1243c35e2"
format_stamp: "Formatted at 2026-08-06 09:12:02 on dist-test-slave-w4v5"
server_key: "152092a68a5e01353ae01ced8cf7b73f"
server_key_iv: "0fd8e396c89b4c5ec5fc6668aab9283d"
server_key_version: "encryptionkey@0"
I20260806 09:12:02.128585 18010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "5dce315adf2748b09db555e1243c35e2"
format_stamp: "Formatted at 2026-08-06 09:12:02 on dist-test-slave-w4v5"
server_key: "152092a68a5e01353ae01ced8cf7b73f"
server_key_iv: "0fd8e396c89b4c5ec5fc6668aab9283d"
server_key_version: "encryptionkey@0"
I20260806 09:12:02.134042 18010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:12:02.137522 18030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:02.138813 18010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:12:02.138998 18010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "5dce315adf2748b09db555e1243c35e2"
format_stamp: "Formatted at 2026-08-06 09:12:02 on dist-test-slave-w4v5"
server_key: "152092a68a5e01353ae01ced8cf7b73f"
server_key_iv: "0fd8e396c89b4c5ec5fc6668aab9283d"
server_key_version: "encryptionkey@0"
I20260806 09:12:02.139292 18010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:12:02.153437 18010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:12:02.157260 18010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:12:02.157565 18010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:12:02.158502 18010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:12:02.159929 18010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:12:02.160059 18010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:02.160215 18010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:12:02.160297 18010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:02.195622 18010 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:42993
I20260806 09:12:02.195731 18187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:42993 every 8 connection(s)
I20260806 09:12:02.196951 18010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:12:02.202553 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 18010
I20260806 09:12:02.202786 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:12:02.203616 3348 external_mini_cluster.cc:1428] Setting key 3f0ab88ca0742b1f10ca36c7a6dd9d15
Aug 06 09:12:02 dist-test-slave-w4v5 krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:12:02.219987 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:02.199672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:53147 (local address 127.3.69.62:40035)
0806 09:12:02.200160 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:02.200167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:02.201114 (+ 947us) server_negotiation.cc:407] Connection header received
0806 09:12:02.202971 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:02.202977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:02.203069 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:02.203230 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:02.204988 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:02.206070 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:02.207748 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:02.208447 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:02.208603 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:02.213224 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:02.213232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:02.213239 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:02.213283 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:02.216678 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:02.217468 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:02.217475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:02.217481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:02.217651 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:02.218082 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:02.218089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:02.218096 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:02.218431 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:02.218551 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:02.219540 (+ 989us) server_negotiation.cc:299] Negotiation successful
0806 09:12:02.219716 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":359,"thread_start_us":228,"threads_started":1}
I20260806 09:12:02.221611 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:02.200233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40035 (local address 127.3.69.1:53147)
0806 09:12:02.200944 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:02.200981 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:02.202600 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:02.203429 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:02.203437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:02.204009 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:12:02.204813 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:02.204838 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:02.206248 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:02.206252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:02.207578 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:02.207592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:02.208724 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:02.208730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:02.209068 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:02.209853 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:12:02.209884 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:12:02.212983 (+ 3099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:12:02.216869 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:02.216878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:02.216897 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:02.217333 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:02.217773 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:02.217776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:02.217778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:02.217963 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:02.218615 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:12:02.218626 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:12:02.219109 (+ 483us) client_negotiation.cc:769] Sending connection context
0806 09:12:02.219436 (+ 327us) client_negotiation.cc:240] Negotiation successful
0806 09:12:02.220043 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":269,"threads_started":1}
I20260806 09:12:02.223516 18188 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40035
I20260806 09:12:02.223866 18188 heartbeater.cc:464] Registering TS with master...
I20260806 09:12:02.224640 18188 heartbeater.cc:511] Master 127.3.69.62:40035 requested a full tablet report, sending...
I20260806 09:12:02.227059 17931 ts_manager.cc:194] Registered new tserver with Master: 5dce315adf2748b09db555e1243c35e2 (127.3.69.1:42993)
I20260806 09:12:02.229318 17931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:53147
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:12:02.269770 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40035
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:12:02.610050 18195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:12:02.610347 18195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:12:02.610519 18195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:12:02.617654 18195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:12:02.617754 18195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:12:02.617928 18195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:12:02.627463 18195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:12:02.628692 18195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:12:02.629997 18195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:12:02.643234 18203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:02.643296 18206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:02.643258 18208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:12:02.644299 18195 server_base.cc:1034] running on GCE node
I20260806 09:12:02.644887 18195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:12:02.645864 18195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:12:02.647142 18195 hybrid_clock.cc:648] HybridClock initialized: now 1786007522647061 us; error 49 us; skew 500 ppm
I20260806 09:12:02.647459 18195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:12:02 dist-test-slave-w4v5 krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:02.650560 18195 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:12:02.652012 18195 webserver.cc:460] Webserver started at http://127.3.69.2:45527/ using document root <none> and password file <none>
I20260806 09:12:02.652688 18195 fs_manager.cc:359] Metadata directory not provided
I20260806 09:12:02.652837 18195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:12:02.653147 18195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:12:02.656165 18195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "f3975f8798da487196bdc93f9d8159b6"
format_stamp: "Formatted at 2026-08-06 09:12:02 on dist-test-slave-w4v5"
server_key: "efd6fa1390bb9a4a19329cc01315c026"
server_key_iv: "8a4d7118d97ea5ce92eeed236c531996"
server_key_version: "encryptionkey@0"
I20260806 09:12:02.656957 18195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "f3975f8798da487196bdc93f9d8159b6"
format_stamp: "Formatted at 2026-08-06 09:12:02 on dist-test-slave-w4v5"
server_key: "efd6fa1390bb9a4a19329cc01315c026"
server_key_iv: "8a4d7118d97ea5ce92eeed236c531996"
server_key_version: "encryptionkey@0"
I20260806 09:12:02.662070 18195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:12:02.665485 18217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:02.666885 18195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:12:02.667054 18195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "f3975f8798da487196bdc93f9d8159b6"
format_stamp: "Formatted at 2026-08-06 09:12:02 on dist-test-slave-w4v5"
server_key: "efd6fa1390bb9a4a19329cc01315c026"
server_key_iv: "8a4d7118d97ea5ce92eeed236c531996"
server_key_version: "encryptionkey@0"
I20260806 09:12:02.667310 18195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:12:02.708159 18195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:12:02.713016 18195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:12:02.713344 18195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:12:02.714313 18195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:12:02.715764 18195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:12:02.715839 18195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:02.716013 18195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:12:02.716071 18195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:02.752027 18195 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:37025
I20260806 09:12:02.752092 18376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:37025 every 8 connection(s)
I20260806 09:12:02.753334 18195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:12:02.757496 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 18195
I20260806 09:12:02.757720 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:12:02.758322 3348 external_mini_cluster.cc:1428] Setting key c5fcd039ba91b0603318b6ea393fea0c
Aug 06 09:12:02 dist-test-slave-w4v5 krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:12:02.779002 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:02.756077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:53269 (local address 127.3.69.62:40035)
0806 09:12:02.756567 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:02.756575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:02.759051 (+ 2476us) server_negotiation.cc:407] Connection header received
0806 09:12:02.760918 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:02.760924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:02.761050 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:02.761181 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:02.763434 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:02.764690 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:02.767372 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:02.768118 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:02.768271 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:02.772305 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:02.772310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:02.772314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:02.772349 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:02.775629 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:02.776373 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:02.776381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:02.776388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:02.776548 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:02.776985 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:02.776993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:02.776999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:02.777384 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:02.777540 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:02.778216 (+ 676us) server_negotiation.cc:299] Negotiation successful
0806 09:12:02.778488 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":224,"threads_started":1}
I20260806 09:12:02.779608 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:02.756651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40035 (local address 127.3.69.2:53269)
0806 09:12:02.757791 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:02.757875 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:02.760551 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:02.761533 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:02.761544 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:02.762302 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:12:02.763231 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:02.763259 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:02.764877 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:02.764882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:02.767218 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:02.767232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:02.768332 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:02.768337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:02.768536 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:02.769288 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:12:02.769311 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:12:02.772048 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:12:02.775786 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:02.775793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:02.775806 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:02.776232 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:02.776660 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:02.776663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:02.776666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:02.776850 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:02.777511 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:12:02.777517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:12:02.777816 (+ 299us) client_negotiation.cc:769] Sending connection context
0806 09:12:02.778037 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 09:12:02.778424 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":425,"threads_started":1}
I20260806 09:12:02.781103 18377 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40035
I20260806 09:12:02.781464 18377 heartbeater.cc:464] Registering TS with master...
I20260806 09:12:02.782388 18377 heartbeater.cc:511] Master 127.3.69.62:40035 requested a full tablet report, sending...
I20260806 09:12:02.784159 17931 ts_manager.cc:194] Registered new tserver with Master: f3975f8798da487196bdc93f9d8159b6 (127.3.69.2:37025)
I20260806 09:12:02.785005 17931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:53269
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:12:02.824467 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40035
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:12:03.165277 18385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:12:03.165545 18385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:12:03.165727 18385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:12:03.173346 18385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:12:03.173457 18385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:12:03.173671 18385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:12:03.183475 18385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:12:03.184826 18385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:12:03.186167 18385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:12:03.199337 18395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:03.199302 18393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:03.200048 18397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:12:03.200552 18385 server_base.cc:1034] running on GCE node
I20260806 09:12:03.201278 18385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:12:03.202131 18385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:12:03.203372 18385 hybrid_clock.cc:648] HybridClock initialized: now 1786007523203313 us; error 47 us; skew 500 ppm
I20260806 09:12:03.203688 18385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:12:03 dist-test-slave-w4v5 krb5kdc[17867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:03.209827 18385 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:12:03.211369 18385 webserver.cc:460] Webserver started at http://127.3.69.3:42511/ using document root <none> and password file <none>
I20260806 09:12:03.212049 18385 fs_manager.cc:359] Metadata directory not provided
I20260806 09:12:03.212210 18385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:12:03.212524 18385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:12:03.215435 18385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "79ba11ceb5d141a09078990b2da352d9"
format_stamp: "Formatted at 2026-08-06 09:12:03 on dist-test-slave-w4v5"
server_key: "7704e22458ea6046803afb85270b986a"
server_key_iv: "624b33586962fc08452b4d202d38189e"
server_key_version: "encryptionkey@0"
I20260806 09:12:03.216215 18385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "79ba11ceb5d141a09078990b2da352d9"
format_stamp: "Formatted at 2026-08-06 09:12:03 on dist-test-slave-w4v5"
server_key: "7704e22458ea6046803afb85270b986a"
server_key_iv: "624b33586962fc08452b4d202d38189e"
server_key_version: "encryptionkey@0"
I20260806 09:12:03.221204 18385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:12:03.224606 18408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:03.225917 18385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:12:03.226109 18385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "79ba11ceb5d141a09078990b2da352d9"
format_stamp: "Formatted at 2026-08-06 09:12:03 on dist-test-slave-w4v5"
server_key: "7704e22458ea6046803afb85270b986a"
server_key_iv: "624b33586962fc08452b4d202d38189e"
server_key_version: "encryptionkey@0"
I20260806 09:12:03.226316 18385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:12:03.233435 18188 heartbeater.cc:503] Master 127.3.69.62:40035 was elected leader, sending a full tablet report...
I20260806 09:12:03.243216 18385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:12:03.247327 18385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:12:03.247617 18385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:12:03.248596 18385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:12:03.249958 18385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:12:03.250072 18385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:03.250227 18385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:12:03.250300 18385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:03.287880 18385 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:32849
I20260806 09:12:03.288040 18573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:32849 every 8 connection(s)
I20260806 09:12:03.289206 18385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:12:03.292819 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 18385
I20260806 09:12:03.293035 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:12:03.293601 3348 external_mini_cluster.cc:1428] Setting key 5d2ec80e72c04a6caa10d1af0d21b240
Aug 06 09:12:03 dist-test-slave-w4v5 krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:12:03.311699 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.291858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:40561 (local address 127.3.69.62:40035)
0806 09:12:03.292338 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.292343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.293422 (+ 1079us) server_negotiation.cc:407] Connection header received
0806 09:12:03.295920 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.295924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.295985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.296070 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:03.297970 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.298736 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.300431 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.300820 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.300913 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.304819 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:03.304824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:03.304827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:03.304861 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:03.308347 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.309307 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.309313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.309316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.309483 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.309922 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.309927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.309931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.310232 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:03.310343 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.311216 (+ 873us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.311378 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260806 09:12:03.312678 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.292247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40035 (local address 127.3.69.3:40561)
0806 09:12:03.293243 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.293285 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:03.295460 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:03.296359 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:03.296370 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:03.297109 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:12:03.297812 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.297831 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.298931 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.298935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.300294 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.300306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.301034 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.301039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.301243 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.301958 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:12:03.301982 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:12:03.304607 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:12:03.308513 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:03.308521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:03.308535 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:03.309163 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:03.309604 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:03.309608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:03.309610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:03.309793 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:03.310356 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:12:03.310362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:12:03.310797 (+ 435us) client_negotiation.cc:769] Sending connection context
0806 09:12:03.311049 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 09:12:03.311521 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":382,"threads_started":1}
I20260806 09:12:03.314085 18574 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40035
I20260806 09:12:03.314369 18574 heartbeater.cc:464] Registering TS with master...
I20260806 09:12:03.315346 18574 heartbeater.cc:511] Master 127.3.69.62:40035 requested a full tablet report, sending...
I20260806 09:12:03.316762 17931 ts_manager.cc:194] Registered new tserver with Master: 79ba11ceb5d141a09078990b2da352d9 (127.3.69.3:32849)
I20260806 09:12:03.317586 17931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:40561
I20260806 09:12:03.321885 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:12:03.340849 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.326779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.3.69.62:40035)
0806 09:12:03.327012 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.327021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.327535 (+ 514us) server_negotiation.cc:407] Connection header received
0806 09:12:03.327648 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.327652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.327725 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.327805 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:03.329372 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.330280 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.331949 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.332325 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.332422 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.334727 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:03.334731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:03.334735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:03.334766 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:03.337695 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.338365 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.338369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.338371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.338541 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.338880 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.338883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.338885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.339726 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:03.339991 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.340268 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.340472 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:12:03.345242 17931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:12:03.352279 17931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:12:03.377810 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.366483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.3.69.2:37025)
0806 09:12:03.368304 (+ 1821us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.368310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.368333 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:12:03.368405 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.368410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.368578 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.368812 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:03.369635 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.371509 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.374701 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.376288 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.376434 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.376855 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.377128 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.377398 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":1528,"threads_started":1}
I20260806 09:12:03.377851 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.366503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:37025 (local address 127.0.0.1:39632)
0806 09:12:03.367540 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.367575 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:03.367751 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:03.368824 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:03.368831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:03.368861 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:03.369406 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.369421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.371736 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.371744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.374482 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.374495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.376583 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.376591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.376858 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.376916 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 09:12:03.377064 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 09:12:03.377302 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":1,"thread_start_us":2678,"threads_started":1}
I20260806 09:12:03.380118 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.366806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:32849 (local address 127.0.0.1:58266)
0806 09:12:03.369495 (+ 2689us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.369514 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:03.369617 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:03.370821 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:03.370826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:03.370852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:03.371852 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.371866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.373908 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.373913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.375856 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.375868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.379288 (+ 3420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.379300 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.379516 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.379555 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:12:03.379633 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 09:12:03.379781 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2543,"thread_start_us":393,"threads_started":1}
I20260806 09:12:03.380892 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.367149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.3.69.3:32849)
0806 09:12:03.370186 (+ 3037us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.370195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.370227 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:12:03.370323 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.370330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.370538 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.371008 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:03.372117 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.373764 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.376017 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.377712 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.377849 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.378195 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.380403 (+ 2208us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.380579 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2825,"thread_start_us":2662,"threads_started":1}
I20260806 09:12:03.382282 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.370864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41074)
0806 09:12:03.372135 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.372156 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:03.372291 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:03.372859 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:03.372863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:03.372886 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:03.373436 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.373446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.375390 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.375395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.379374 (+ 3979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.379395 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.381441 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.381447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.381665 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.381699 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:12:03.381819 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 09:12:03.381955 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"thread_start_us":1524,"threads_started":1}
I20260806 09:12:03.382679 18284 tablet_service.cc:1467] Processing CreateTablet for tablet d2b267eca0dd48628f814577a5379cfa (DEFAULT_TABLE table=test-table [id=bf7e69a913f940ecbbe24d7518c3db12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:12:03.383728 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.370058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.3.69.1:42993)
0806 09:12:03.372149 (+ 2091us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.372159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.372403 (+ 244us) server_negotiation.cc:407] Connection header received
0806 09:12:03.372497 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.372504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.372683 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.372865 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:03.373637 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.375243 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.379607 (+ 4364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.381225 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.381376 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.381681 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.383234 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.383415 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1914,"thread_start_us":2129,"threads_started":1}
I20260806 09:12:03.384147 18284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b267eca0dd48628f814577a5379cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:12:03.384307 18478 tablet_service.cc:1467] Processing CreateTablet for tablet d2b267eca0dd48628f814577a5379cfa (DEFAULT_TABLE table=test-table [id=bf7e69a913f940ecbbe24d7518c3db12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:12:03.385556 18478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b267eca0dd48628f814577a5379cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:12:03.387790 18097 tablet_service.cc:1467] Processing CreateTablet for tablet d2b267eca0dd48628f814577a5379cfa (DEFAULT_TABLE table=test-table [id=bf7e69a913f940ecbbe24d7518c3db12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:12:03.389322 18097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b267eca0dd48628f814577a5379cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:12:03.398819 18602 tablet_bootstrap.cc:492] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: Bootstrap starting.
I20260806 09:12:03.400897 18603 tablet_bootstrap.cc:492] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Bootstrap starting.
I20260806 09:12:03.402529 18602 tablet_bootstrap.cc:654] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: Neither blocks nor log segments found. Creating new log.
I20260806 09:12:03.404023 18602 log.cc:824] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:12:03.404541 18604 tablet_bootstrap.cc:492] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: Bootstrap starting.
I20260806 09:12:03.404830 18603 tablet_bootstrap.cc:654] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Neither blocks nor log segments found. Creating new log.
I20260806 09:12:03.406667 18603 log.cc:824] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Log is configured to *not* fsync() on all Append() calls
I20260806 09:12:03.408414 18602 tablet_bootstrap.cc:492] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: No bootstrap required, opened a new log
I20260806 09:12:03.408828 18602 ts_tablet_manager.cc:1397] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 09:12:03.408905 18604 tablet_bootstrap.cc:654] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: Neither blocks nor log segments found. Creating new log.
I20260806 09:12:03.409891 18604 log.cc:824] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: Log is configured to *not* fsync() on all Append() calls
I20260806 09:12:03.410447 18603 tablet_bootstrap.cc:492] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: No bootstrap required, opened a new log
I20260806 09:12:03.410789 18603 ts_tablet_manager.cc:1397] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 09:12:03.412688 18604 tablet_bootstrap.cc:492] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: No bootstrap required, opened a new log
I20260806 09:12:03.412952 18604 ts_tablet_manager.cc:1397] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260806 09:12:03.421146 18604 raft_consensus.cc:348] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.421569 18604 raft_consensus.cc:374] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:12:03.421677 18604 raft_consensus.cc:729] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dce315adf2748b09db555e1243c35e2, State: Initialized, Role: FOLLOWER
I20260806 09:12:03.421663 18602 raft_consensus.cc:348] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.422245 18602 raft_consensus.cc:374] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:12:03.422376 18602 raft_consensus.cc:729] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba11ceb5d141a09078990b2da352d9, State: Initialized, Role: FOLLOWER
I20260806 09:12:03.422361 18604 consensus_queue.cc:260] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.422876 18603 raft_consensus.cc:348] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.423043 18602 consensus_queue.cc:260] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.423400 18603 raft_consensus.cc:374] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:12:03.424508 18574 heartbeater.cc:503] Master 127.3.69.62:40035 was elected leader, sending a full tablet report...
I20260806 09:12:03.424508 18604 ts_tablet_manager.cc:1428] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260806 09:12:03.424871 18603 raft_consensus.cc:729] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3975f8798da487196bdc93f9d8159b6, State: Initialized, Role: FOLLOWER
I20260806 09:12:03.425071 18602 ts_tablet_manager.cc:1428] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 09:12:03.425577 18603 consensus_queue.cc:260] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.427315 18377 heartbeater.cc:503] Master 127.3.69.62:40035 was elected leader, sending a full tablet report...
I20260806 09:12:03.427561 18603 ts_tablet_manager.cc:1428] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260806 09:12:03.444685 18614 raft_consensus.cc:482] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:12:03.445040 18614 raft_consensus.cc:504] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.446617 18614 leader_election.cc:288] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3975f8798da487196bdc93f9d8159b6 (127.3.69.2:37025), 5dce315adf2748b09db555e1243c35e2 (127.3.69.1:42993)
W20260806 09:12:03.449424 18189 tablet.cc:2369] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:12:03.454440 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.447248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:35819 (local address 127.3.69.2:37025)
0806 09:12:03.447481 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.447489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.447511 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:12:03.447669 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.447677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.447802 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.447908 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:03.448769 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.449878 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.451926 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.453526 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.453660 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.453735 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.454075 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.454171 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:12:03.454461 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.447146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:37025 (local address 127.3.69.3:35819)
0806 09:12:03.447390 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.447407 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:03.447532 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:03.448106 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:03.448112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:03.448151 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:03.448554 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.448572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.450024 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.450030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.451753 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.451766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.453708 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.453714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.453906 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.453941 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:12:03.454045 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 09:12:03.454182 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 09:12:03.455703 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.447608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.3.69.3:54405)
0806 09:12:03.448573 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.448591 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:03.448748 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:03.449098 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:03.449102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:03.449127 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:03.449554 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.449563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.451165 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.451169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.453248 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:03.453264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.454940 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.454944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:03.455120 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.455158 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:12:03.455257 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:12:03.455388 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":3,"thread_start_us":429,"threads_started":1}
I20260806 09:12:03.456312 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.447791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:54405 (local address 127.3.69.1:42993)
0806 09:12:03.447999 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.448006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.448685 (+ 679us) server_negotiation.cc:407] Connection header received
0806 09:12:03.448905 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.448917 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.448975 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.449074 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:03.449752 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.450908 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.453418 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.454572 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.454725 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.454835 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.455307 (+ 472us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.456006 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:12:03.456199 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b267eca0dd48628f814577a5379cfa" candidate_uuid: "79ba11ceb5d141a09078990b2da352d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3975f8798da487196bdc93f9d8159b6" is_pre_election: true
I20260806 09:12:03.456808 18310 raft_consensus.cc:2454] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ba11ceb5d141a09078990b2da352d9 in term 0.
I20260806 09:12:03.457345 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b267eca0dd48628f814577a5379cfa" candidate_uuid: "79ba11ceb5d141a09078990b2da352d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dce315adf2748b09db555e1243c35e2" is_pre_election: true
I20260806 09:12:03.457605 18410 leader_election.cc:302] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba11ceb5d141a09078990b2da352d9, f3975f8798da487196bdc93f9d8159b6; no voters:
I20260806 09:12:03.457850 18125 raft_consensus.cc:2454] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ba11ceb5d141a09078990b2da352d9 in term 0.
I20260806 09:12:03.458047 18614 raft_consensus.cc:2781] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:12:03.458292 18614 raft_consensus.cc:482] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:12:03.458488 18614 raft_consensus.cc:3037] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:12:03.461223 18614 raft_consensus.cc:504] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.461994 18614 leader_election.cc:288] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [CANDIDATE]: Term 1 election: Requested vote from peers f3975f8798da487196bdc93f9d8159b6 (127.3.69.2:37025), 5dce315adf2748b09db555e1243c35e2 (127.3.69.1:42993)
I20260806 09:12:03.462328 18310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b267eca0dd48628f814577a5379cfa" candidate_uuid: "79ba11ceb5d141a09078990b2da352d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3975f8798da487196bdc93f9d8159b6"
I20260806 09:12:03.462419 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b267eca0dd48628f814577a5379cfa" candidate_uuid: "79ba11ceb5d141a09078990b2da352d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dce315adf2748b09db555e1243c35e2"
I20260806 09:12:03.462520 18310 raft_consensus.cc:3037] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:12:03.462671 18125 raft_consensus.cc:3037] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:12:03.464998 18125 raft_consensus.cc:2454] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba11ceb5d141a09078990b2da352d9 in term 1.
I20260806 09:12:03.465415 18410 leader_election.cc:302] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dce315adf2748b09db555e1243c35e2, 79ba11ceb5d141a09078990b2da352d9; no voters:
I20260806 09:12:03.465495 18310 raft_consensus.cc:2454] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba11ceb5d141a09078990b2da352d9 in term 1.
I20260806 09:12:03.465888 18614 raft_consensus.cc:2781] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:12:03.466498 18614 raft_consensus.cc:686] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 1 LEADER]: Becoming Leader. State: Replica: 79ba11ceb5d141a09078990b2da352d9, State: Running, Role: LEADER
I20260806 09:12:03.467034 18614 consensus_queue.cc:237] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } }
I20260806 09:12:03.471903 17929 catalog_manager.cc:5495] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79ba11ceb5d141a09078990b2da352d9 (127.3.69.3). New cstate: current_term: 1 leader_uuid: "79ba11ceb5d141a09078990b2da352d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba11ceb5d141a09078990b2da352d9" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 32849 } health_report { overall_health: HEALTHY } } }
W20260806 09:12:03.505491 18378 tablet.cc:2369] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:12:03.513759 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.507354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.3.69.3:32849)
0806 09:12:03.507577 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.507585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.507709 (+ 124us) server_negotiation.cc:407] Connection header received
0806 09:12:03.507955 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.507959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.508049 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.508216 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:12:03.508639 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.509401 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.511557 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.511980 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.512087 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.512647 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:12:03.513173 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:12:03.513283 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.513440 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.513511 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:12:03.523406 18125 raft_consensus.cc:1261] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 1 FOLLOWER]: Refusing update from remote peer 79ba11ceb5d141a09078990b2da352d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:12:03.523545 18310 raft_consensus.cc:1261] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 1 FOLLOWER]: Refusing update from remote peer 79ba11ceb5d141a09078990b2da352d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:12:03.524616 18618 consensus_queue.cc:1035] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dce315adf2748b09db555e1243c35e2" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:12:03.525099 18614 consensus_queue.cc:1035] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3975f8798da487196bdc93f9d8159b6" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:12:03.543464 18631 mvcc.cc:205] Tried to move back new op lower bound from 7315486816341291008 to 7315486816125206528. Current Snapshot: MvccSnapshot[applied={T|T < 7315486816341291008}]
I20260806 09:12:03.557582 17929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:12:03.557904 17929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:12:03.562008 17929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a16914c3b474cdaa05ba1dc3bd6d531: Sending DeleteTablet for 3 replicas of tablet d2b267eca0dd48628f814577a5379cfa
I20260806 09:12:03.563292 18284 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b267eca0dd48628f814577a5379cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:12:03 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:39632
I20260806 09:12:03.563335 18097 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b267eca0dd48628f814577a5379cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:12:03 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:41074
I20260806 09:12:03.563802 18478 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b267eca0dd48628f814577a5379cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:12:03 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:58266
I20260806 09:12:03.566365 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 18010
I20260806 09:12:03.578778 18648 tablet_replica.cc:330] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: stopping tablet replica
I20260806 09:12:03.579491 18648 raft_consensus.cc:2229] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:12:03.580492 18648 raft_consensus.cc:2258] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:12:03.586230 18647 tablet_replica.cc:330] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: stopping tablet replica
I20260806 09:12:03.586926 18647 raft_consensus.cc:2229] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:12:03.587357 18648 ts_tablet_manager.cc:1905] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:12:03.587740 18647 raft_consensus.cc:2258] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:12:03.591055 18646 tablet_replica.cc:330] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: stopping tablet replica
I20260806 09:12:03.591498 18646 raft_consensus.cc:2229] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:12:03.591935 18646 raft_consensus.cc:2258] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:12:03.593520 18648 ts_tablet_manager.cc:1918] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:12:03.593710 18648 log.cc:1196] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/d2b267eca0dd48628f814577a5379cfa
I20260806 09:12:03.594336 18648 ts_tablet_manager.cc:1939] T d2b267eca0dd48628f814577a5379cfa P 79ba11ceb5d141a09078990b2da352d9: Deleting consensus metadata
Aug 06 09:12:03 dist-test-slave-w4v5 krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:12:03.596434 18647 ts_tablet_manager.cc:1905] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:12:03.596416 17912 catalog_manager.cc:4841] TS 79ba11ceb5d141a09078990b2da352d9 (127.3.69.3:32849): tablet d2b267eca0dd48628f814577a5379cfa (table test-table [id=bf7e69a913f940ecbbe24d7518c3db12]) successfully deleted
I20260806 09:12:03.600689 18646 ts_tablet_manager.cc:1905] T d2b267eca0dd48628f814577a5379cfa P 5dce315adf2748b09db555e1243c35e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:12:03.604255 18647 ts_tablet_manager.cc:1918] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:12:03.604498 18647 log.cc:1196] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/d2b267eca0dd48628f814577a5379cfa
I20260806 09:12:03.604665 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.569366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.3.69.1:42993)
0806 09:12:03.569624 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.569634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.574142 (+ 4508us) server_negotiation.cc:407] Connection header received
0806 09:12:03.575617 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.575627 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.575816 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.575916 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:03.581050 (+ 5134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.582667 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.588271 (+ 5604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.589291 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.589426 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.596828 (+ 7402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:03.596843 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:03.596860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:03.596904 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:03.600782 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.601671 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.601684 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.601702 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.601883 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.602377 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.602386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.602390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.602941 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:03.603399 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.603978 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.604237 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:12:03.605391 18647 ts_tablet_manager.cc:1939] T d2b267eca0dd48628f814577a5379cfa P f3975f8798da487196bdc93f9d8159b6: Deleting consensus metadata
I20260806 09:12:03.606176 18175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:12:03.607512 17911 catalog_manager.cc:4841] TS f3975f8798da487196bdc93f9d8159b6 (127.3.69.2:37025): tablet d2b267eca0dd48628f814577a5379cfa (table test-table [id=bf7e69a913f940ecbbe24d7518c3db12]) successfully deleted
I20260806 09:12:03.697777 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 18010
W20260806 09:12:03.727124 17911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.69.1:42993 (error 108)
I20260806 09:12:03.729048 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.728607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41100)
0806 09:12:03.728802 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.728906 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 09:12:03.729045 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 18195
W20260806 09:12:03.729645 17911 catalog_manager.cc:4584] TS 5dce315adf2748b09db555e1243c35e2 (127.3.69.1:42993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2b267eca0dd48628f814577a5379cfa: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Aug 06 09:12:03 dist-test-slave-w4v5 krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:12:03.745400 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.730114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.3.69.2:37025)
0806 09:12:03.730296 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.730303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.730532 (+ 229us) server_negotiation.cc:407] Connection header received
0806 09:12:03.730718 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.730724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.730810 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.730924 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:03.732415 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.733183 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.734527 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.734920 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.735069 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.739570 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:03.739584 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:03.739599 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:03.739637 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:03.742568 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.743310 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.743326 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.743329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.743455 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.743847 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.743851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.743853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.744220 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:03.744525 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.744859 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.745084 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:12:03.746091 18371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:12:03.783595 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.783012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41114)
0806 09:12:03.783276 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.783399 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 09:12:03.784834 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.784393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41124)
0806 09:12:03.784571 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.784664 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 09:12:03.836647 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 18195
I20260806 09:12:03.845264 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.844792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41132)
0806 09:12:03.845000 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.845096 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 09:12:03.846737 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.846267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41146)
0806 09:12:03.846511 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.846607 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260806 09:12:03.865833 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 18385
Aug 06 09:12:03 dist-test-slave-w4v5 krb5kdc[17867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:12:03.882965 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.866738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.3.69.3:32849)
0806 09:12:03.866971 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:03.866983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:03.867009 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:12:03.867191 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:03.867202 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:03.867281 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:03.867391 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:03.868806 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.869672 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.871002 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:03.871718 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:03.871866 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:03.876602 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:03.876613 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:03.876628 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:03.876668 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:03.879846 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.880627 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.880637 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.880641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.880868 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:03.881250 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:03.881255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:03.881259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:03.881733 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:03.882058 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:03.882276 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 09:12:03.882548 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 09:12:03.883795 18566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:12:03.928045 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.927466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41162)
0806 09:12:03.927741 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.927856 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260806 09:12:03.929309 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:03.928828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41166)
0806 09:12:03.929076 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:03.929170 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260806 09:12:04.006851 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 18385
I20260806 09:12:04.036171 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 17887
I20260806 09:12:04.037420 17966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:12:04.145483 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:04.144888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42993 (local address 127.0.0.1:41182)
0806 09:12:04.145085 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:04.145205 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 09:12:04.145730 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 17887
2026-08-06T09:12:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4683 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:12:04 dist-test-slave-w4v5 krb5kdc[18661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:12:04 dist-test-slave-w4v5 krb5kdc[18661](info): set up 2 sockets
Aug 06 09:12:04 dist-test-slave-w4v5 krb5kdc[18661](info): commencing operation
krb5kdc: starting...
W20260806 09:12:05.460549 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:12:05 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:12:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.69.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.69.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/rangeradmin_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/rangeradmin_127.3.69.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.69.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.69.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/rangerlookup_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/rangerlookup_127.3.69.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.69.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.69.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/HTTP_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/HTTP_127.3.69.20@KRBTEST.COM.keytab.
I20260806 09:12:05.837702 3348 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6yYR2E/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/postgres -l logfile start
2026-08-06 09:12:08.353 UTC [18717] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 09:12:08.353 UTC [18717] LOG: listening on IPv4 address "127.3.69.20", port 39415
2026-08-06 09:12:08.357 UTC [18717] LOG: listening on Unix socket "/tmp/.s.PGSQL.39415"
2026-08-06 09:12:08.381 UTC [18720] LOG: database system was shut down at 2026-08-06 09:12:06 UTC
2026-08-06 09:12:08.385 UTC [18717] LOG: database system is ready to accept connections
I20260806 09:12:09.537766 3348 mini_postgres.cc:96] Postgres bound to 39415
2026-08-06 09:12:09.562 UTC [18735] FATAL: database "slave" does not exist
127.3.69.20:39415 - accepting connections
I20260806 09:12:09.563025 3348 mini_ranger.cc:161] Starting Ranger...
I20260806 09:12:09.593672 3348 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 09:12:10.072692 3348 mini_ranger.cc:87] Created ranger Postgres database
I20260806 09:12:10.072876 3348 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-admin
2026-08-06 09:12:10,420 [I] DB FLAVOR :POSTGRES
2026-08-06 09:12:10,420 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:12:10,420 [I] Checking connection..
2026-08-06 09:12:10,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 09:12:10,679 [I] Checking connection passed.
2026-08-06 09:12:10,679 [I] --------- Verifying version history table ---------
2026-08-06 09:12:10,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:12:10,923 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:12:10,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:12:11,189 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:12:11,190 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 09:12:11,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 09:12:11.434 UTC [18827] WARNING: there is no transaction in progress
2026-08-06 09:12:11,441 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 09:12:11,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:12:11,713 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 09:12:11,713 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:12:11,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 09:12:11,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 09:12:12,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w4v5.c.gcp-upstream.internal','N') ;"
2026-08-06 09:12:12,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:12:12,734 [I] Table x_portal_user does not exist in database ranger
2026-08-06 09:12:12,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:12:13,006 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 09:12:13,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:12:13,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:12:13,528 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 09:12:13,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 09:12:14.469 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.485 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.501 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.516 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.531 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.552 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.607 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.614 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.628 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.637 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.647 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.660 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.670 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.676 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.682 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:14.689 UTC [18985] WARNING: there is no transaction in progress
2026-08-06 09:12:15,335 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 09:12:15,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 09:12:15,598 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 09:12:15,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:12:15,846 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 09:12:15,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:12:16,112 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 09:12:16,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:12:16,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:12:16,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w4v5.c.gcp-upstream.internal';"
2026-08-06 09:12:16,850 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 09:12:16,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 09:12:17,112 [I] DB_PATCHES have already been applied
I20260806 09:12:17.119134 3348 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-admin:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6yYR2E/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 09:12:17.119256 3348 mini_ranger.cc:193] Using host: 127.3.69.20
I20260806 09:12:17.140699 3348 mini_ranger.cc:227] Ranger admin URL: http://127.3.69.20:57843
Aug 06, 2026 9:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 9:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 9:12:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 9:12:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 9:12:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 09:12:18.536094 3348 mini_ranger.cc:160] Time spent starting Ranger: real 8.973s user 0.007s sys 0.050s
Aug 06, 2026 9:12:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 9:12:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 9:12:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 9:12:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 9:12:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 09:12:40.784377 3348 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.69.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.69.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/rangerkms_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/rangerkms_127.3.69.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.69.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.69.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.69.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/HTTP_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.69.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/HTTP_127.3.69.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 09:12:41 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007561, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 09:12:41.015486 3348 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 09:12:41.042351 3348 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 09:12:41.904023 3348 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 09:12:41.904214 3348 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms
I20260806 09:12:41.904359 3348 mini_ranger_kms.cc:227] Using postgres at 127.3.69.20:39415
2026-08-06 09:12:42,532 [I] DB FLAVOR :POSTGRES
2026-08-06 09:12:42,533 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:12:42,533 [I] Checking connection
2026-08-06 09:12:42,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 09:12:42,799 [I] connection success
2026-08-06 09:12:42,799 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 09:12:42,800 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 09:12:42,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 09:12:43,066 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 09:12:43,066 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:12:43,066 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 09:12:43,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:39415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 09:12:43,375 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 09:12:43.834074 3348 mini_ranger_kms.cc:310] Created kms service
I20260806 09:12:44.036521 3348 mini_ranger_kms.cc:326] Created kudu user
I20260806 09:12:44.084035 3348 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 09:12:44 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007561, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:12:44.507432 3348 mini_ranger_kms.cc:384] Added ranger policy
I20260806 09:12:44.507783 3348 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms:/tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 09:12:44.507829 3348 mini_ranger_kms.cc:240] Using host: 127.3.69.20
I20260806 09:12:44.537703 3348 mini_ranger_kms.cc:276] Ranger KMS PID: 19244
I20260806 09:12:44.537928 3348 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.69.20:40461
Aug 06, 2026 9:12:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:12:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:12:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 9:12:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 9:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 9:12:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 09:12:46 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007566, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.69.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:47.230019 3348 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.215s user 0.159s sys 0.492s
I20260806 09:12:47.230389 3348 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 09:12:47.230643 3348 mini_ranger_kms.cc:401] 127.3.69.20:40461/kms/v1/keys
Aug 06 09:12:47 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007566, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.69.20@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
Aug 06 09:12:47 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007566, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.69.20@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
Aug 06 09:12:49 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:12:50.059803 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:43989
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:43989
--ranger_config_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:40461/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:12:50.510053 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:12:50.510391 19319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:12:50.510536 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:12:50.520205 19319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:12:50.520363 19319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:12:50.520437 19319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:12:50.520555 19319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:12:50.520640 19319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:12:50.533851 19319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.69.20:40461/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:43989
--ranger_config_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:12:50.535403 19319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:12:50.536871 19319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:12:50.552650 19324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:50.552815 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:50.552999 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:12:50.554065 19319 server_base.cc:1034] running on GCE node
I20260806 09:12:50.555297 19319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:12:50.556982 19319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:12:50.558357 19319 hybrid_clock.cc:648] HybridClock initialized: now 1786007570558241 us; error 38 us; skew 500 ppm
I20260806 09:12:50.558784 19319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:12:50 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007570, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:50.566358 19319 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:12:50.568445 19319 webserver.cc:460] Webserver started at http://127.3.69.62:39729/ using document root <none> and password file <none>
I20260806 09:12:50.569224 19319 fs_manager.cc:359] Metadata directory not provided
I20260806 09:12:50.569519 19319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:12:50.569957 19319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:12:50 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007570, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:12:50.641299 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "709224bd6b9d4641900b0cd7e79f50cd"
format_stamp: "Formatted at 2026-08-06 09:12:50 on dist-test-slave-w4v5"
server_key: "535da630fa4875e348657bd0f915ede0"
server_key_iv: "d51921e91a816d87ec873ee5b7c80124"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:50.642386 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "709224bd6b9d4641900b0cd7e79f50cd"
format_stamp: "Formatted at 2026-08-06 09:12:50 on dist-test-slave-w4v5"
server_key: "535da630fa4875e348657bd0f915ede0"
server_key_iv: "d51921e91a816d87ec873ee5b7c80124"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:50.649801 19319 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260806 09:12:50.671303 19336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:50.672986 19319 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:12:50.673221 19319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "709224bd6b9d4641900b0cd7e79f50cd"
format_stamp: "Formatted at 2026-08-06 09:12:50 on dist-test-slave-w4v5"
server_key: "535da630fa4875e348657bd0f915ede0"
server_key_iv: "d51921e91a816d87ec873ee5b7c80124"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:50.673555 19319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:12:50.692523 19319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:12:50.697786 19319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:12:50.698288 19319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:12:50.747403 19319 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:43989
I20260806 09:12:50.747442 19414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:43989 every 8 connection(s)
I20260806 09:12:50.748991 19319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:12:50.749449 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 19319
I20260806 09:12:50.749732 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:12:50.750782 3348 external_mini_cluster.cc:1428] Setting key 79778c1ad0625fc9624f51fad33fc7ca
I20260806 09:12:50.755781 19416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:12:50 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:12:50.772035 19416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd: Bootstrap starting.
I20260806 09:12:50.776081 19416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd: Neither blocks nor log segments found. Creating new log.
I20260806 09:12:50.777170 19416 log.cc:824] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd: Log is configured to *not* fsync() on all Append() calls
I20260806 09:12:50.779564 19419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:50.753375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.3.69.62:43989)
0806 09:12:50.754545 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:50.754561 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:50.754610 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:12:50.757618 (+ 3008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:50.757696 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:50.758280 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:50.760643 (+ 2363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:50.761407 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:50.763826 (+ 2419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:50.765870 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:50.767247 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:50.767449 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:50.771887 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:50.771895 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:50.771910 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:50.771947 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:50.775010 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:50.775790 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:50.775797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:50.775803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:50.775995 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:50.776458 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:50.776468 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:50.776470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:50.777034 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:50.777413 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:50.777819 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 09:12:50.778302 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":704,"threads_started":1}
I20260806 09:12:50.780972 19416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd: No bootstrap required, opened a new log
I20260806 09:12:50.789443 19416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709224bd6b9d4641900b0cd7e79f50cd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 43989 } }
I20260806 09:12:50.789896 19416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:12:50.790014 19416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709224bd6b9d4641900b0cd7e79f50cd, State: Initialized, Role: FOLLOWER
I20260806 09:12:50.790612 19416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709224bd6b9d4641900b0cd7e79f50cd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 43989 } }
I20260806 09:12:50.790865 19416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:12:50.791010 19416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:12:50.791226 19416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:12:50.793984 19416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709224bd6b9d4641900b0cd7e79f50cd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 43989 } }
I20260806 09:12:50.794742 19416 leader_election.cc:302] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 709224bd6b9d4641900b0cd7e79f50cd; no voters:
I20260806 09:12:50.795670 19416 leader_election.cc:288] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:12:50.795714 19422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:12:50.796918 19422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [term 1 LEADER]: Becoming Leader. State: Replica: 709224bd6b9d4641900b0cd7e79f50cd, State: Running, Role: LEADER
I20260806 09:12:50.797595 19422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709224bd6b9d4641900b0cd7e79f50cd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 43989 } }
I20260806 09:12:50.798403 19416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:12:50.800652 19425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "709224bd6b9d4641900b0cd7e79f50cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709224bd6b9d4641900b0cd7e79f50cd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 43989 } } }
I20260806 09:12:50.801533 19425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [sys.catalog]: This master's current role is: LEADER
I20260806 09:12:50.801033 19426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709224bd6b9d4641900b0cd7e79f50cd. Latest consensus state: current_term: 1 leader_uuid: "709224bd6b9d4641900b0cd7e79f50cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709224bd6b9d4641900b0cd7e79f50cd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 43989 } } }
I20260806 09:12:50.802301 19426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd [sys.catalog]: This master's current role is: LEADER
I20260806 09:12:50.811496 19430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:12:50.816968 19430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:12:50.818032 19416 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 09:12:50.844784 19430 catalog_manager.cc:1347] Generated new cluster ID: d71cc734a9ef45f2b3aa9a34311f4b11
I20260806 09:12:50.845028 19430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:12:50.861923 19430 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:12:50.869059 19430 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:12:50.892923 19430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd: Generated new TSK 0
I20260806 09:12:50.894505 19430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:12:51.791882 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:40461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:43989
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:12:52 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007572, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 09:12:52.342599 19475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:12:52.343024 19475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:12:52.343357 19475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:12:52.355424 19475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:12:52.355690 19475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:12:52.355949 19475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:12:52.371515 19475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.69.20:40461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:12:52.373147 19475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:12:52.374778 19475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:12:52.407254 19493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:52.408118 19496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:52.409833 19491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:12:52.410576 19475 server_base.cc:1034] running on GCE node
I20260806 09:12:52.411381 19475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:12:52.412663 19475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:12:52.414065 19475 hybrid_clock.cc:648] HybridClock initialized: now 1786007572413935 us; error 45 us; skew 500 ppm
I20260806 09:12:52.414603 19475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:12:52 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007572, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:52.425580 19475 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:12:52.436101 19475 webserver.cc:460] Webserver started at http://127.3.69.1:41381/ using document root <none> and password file <none>
I20260806 09:12:52.437191 19475 fs_manager.cc:359] Metadata directory not provided
I20260806 09:12:52.437529 19475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:12:52.438148 19475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:12:52 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007572, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:12:52.465400 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "cc590cb05c984cd38f19175744f406fc"
format_stamp: "Formatted at 2026-08-06 09:12:52 on dist-test-slave-w4v5"
server_key: "aa0e3ac5f9574efe16d822c0a6f90e6c"
server_key_iv: "b31cc103917ca95e26a6c56c345dc8d5"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:52.466747 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "cc590cb05c984cd38f19175744f406fc"
format_stamp: "Formatted at 2026-08-06 09:12:52 on dist-test-slave-w4v5"
server_key: "aa0e3ac5f9574efe16d822c0a6f90e6c"
server_key_iv: "b31cc103917ca95e26a6c56c345dc8d5"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:52.475207 19475 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.005s
I20260806 09:12:52.497659 19506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:52.499682 19475 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260806 09:12:52.499955 19475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "cc590cb05c984cd38f19175744f406fc"
format_stamp: "Formatted at 2026-08-06 09:12:52 on dist-test-slave-w4v5"
server_key: "aa0e3ac5f9574efe16d822c0a6f90e6c"
server_key_iv: "b31cc103917ca95e26a6c56c345dc8d5"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:52.500273 19475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:12:52.538949 19475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:12:52.545387 19475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:12:52.546000 19475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:12:52.548240 19475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:12:52.551797 19475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:12:52.551944 19475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:52.552253 19475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:12:52.552443 19475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:12:52 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007572, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:12:52.994071 19475 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:33295
I20260806 09:12:52.994318 19674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:33295 every 8 connection(s)
I20260806 09:12:52.996719 19475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:12:53.005342 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 19475
I20260806 09:12:53.005638 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:12:53.006942 3348 external_mini_cluster.cc:1428] Setting key 802410efd37d64d43cf208ea8cd32446
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Aug 06 09:12:53 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007572, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:12:53.066455 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:53.001885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:34195 (local address 127.3.69.62:43989)
0806 09:12:53.020538 (+ 18653us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:53.020545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:53.031225 (+ 10680us) server_negotiation.cc:407] Connection header received
0806 09:12:53.035832 (+ 4607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:53.035839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:53.036010 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:53.036155 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:53.042331 (+ 6176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:53.044095 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:53.047488 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:53.048479 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:53.048624 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:53.057249 (+ 8625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:53.057257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:53.057262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:53.057308 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:53.062616 (+ 5308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:53.063769 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:53.063775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:53.063780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:53.063930 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:53.064500 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:53.064505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:53.064508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:53.064806 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:53.064973 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:53.065759 (+ 786us) server_negotiation.cc:299] Negotiation successful
0806 09:12:53.065994 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18459,"thread_start_us":34253,"threads_started":1}
I20260806 09:12:53.067819 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:53.000982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:43989 (local address 127.3.69.1:34195)
0806 09:12:53.030946 (+ 29964us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:53.031042 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:53.035450 (+ 4408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:53.039511 (+ 4061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:53.039535 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:53.040679 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:12:53.042071 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:53.042101 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:53.044337 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:53.044342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:53.047289 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:53.047308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:53.049240 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:53.049249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:53.049666 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:53.050872 (+ 1206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:12:53.050906 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:12:53.056908 (+ 6002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:12:53.062843 (+ 5935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:53.062855 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:53.062871 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:53.063603 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:53.064064 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:53.064069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:53.064077 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:53.064362 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:53.064949 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:12:53.064958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:12:53.065372 (+ 414us) client_negotiation.cc:769] Sending connection context
0806 09:12:53.065655 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 09:12:53.066337 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29489,"thread_start_us":29972,"threads_started":1}
I20260806 09:12:53.069911 19675 heartbeater.cc:347] Connected to a master server at 127.3.69.62:43989
I20260806 09:12:53.070238 19675 heartbeater.cc:464] Registering TS with master...
I20260806 09:12:53.071038 19675 heartbeater.cc:511] Master 127.3.69.62:43989 requested a full tablet report, sending...
I20260806 09:12:53.073302 19361 ts_manager.cc:194] Registered new tserver with Master: cc590cb05c984cd38f19175744f406fc (127.3.69.1:33295)
I20260806 09:12:53.075812 19361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:34195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:12:53.102790 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:40461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:43989
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:12:53 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007572, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
W20260806 09:12:53.830485 19683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:12:53.831022 19683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:12:53.831368 19683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:12:53.841519 19683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:12:53.841722 19683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:12:53.842022 19683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:12:53.856825 19683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.69.20:40461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:12:53.858414 19683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:12:53.860037 19683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:12:53.882583 19700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:53.882635 19701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:53.883009 19704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:12:53.884543 19683 server_base.cc:1034] running on GCE node
I20260806 09:12:53.885293 19683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:12:53.886471 19683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:12:53.887756 19683 hybrid_clock.cc:648] HybridClock initialized: now 1786007573887624 us; error 53 us; skew 500 ppm
I20260806 09:12:53.888154 19683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:12:53 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:53.893182 19683 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:12:53.895905 19683 webserver.cc:460] Webserver started at http://127.3.69.2:42361/ using document root <none> and password file <none>
I20260806 09:12:53.896953 19683 fs_manager.cc:359] Metadata directory not provided
I20260806 09:12:53.897159 19683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:12:53.897740 19683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:12:53 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:12:53.919212 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "8bafa8c243c44e879b47123521431965"
format_stamp: "Formatted at 2026-08-06 09:12:53 on dist-test-slave-w4v5"
server_key: "97287880a9ca614c9de61ca26086d076"
server_key_iv: "51c3f1ffbed5da8aa511d7c32434468c"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:53.920812 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "8bafa8c243c44e879b47123521431965"
format_stamp: "Formatted at 2026-08-06 09:12:53 on dist-test-slave-w4v5"
server_key: "97287880a9ca614c9de61ca26086d076"
server_key_iv: "51c3f1ffbed5da8aa511d7c32434468c"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:53.930123 19683 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260806 09:12:53.952443 19713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:53.954480 19683 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:12:53.954773 19683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "8bafa8c243c44e879b47123521431965"
format_stamp: "Formatted at 2026-08-06 09:12:53 on dist-test-slave-w4v5"
server_key: "97287880a9ca614c9de61ca26086d076"
server_key_iv: "51c3f1ffbed5da8aa511d7c32434468c"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:53.955200 19683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:12:53.992779 19683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:12:54.000308 19683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:12:54.000958 19683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:12:54.003005 19683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:12:54.005903 19683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:12:54.006160 19683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 09:12:54.006457 19683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:12:54.006588 19683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:54.080715 19675 heartbeater.cc:503] Master 127.3.69.62:43989 was elected leader, sending a full tablet report...
I20260806 09:12:54.115002 19683 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:37603
I20260806 09:12:54.115213 19883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:37603 every 8 connection(s)
I20260806 09:12:54.116706 19683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:12:54.122080 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 19683
I20260806 09:12:54.122300 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:12:54.122934 3348 external_mini_cluster.cc:1428] Setting key bd0252aa83e04b66b7cc36884aacfa5c
Aug 06 09:12:54 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007573, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:12:54.153221 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:54.120486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:33153 (local address 127.3.69.62:43989)
0806 09:12:54.121087 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:54.121092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:54.127163 (+ 6071us) server_negotiation.cc:407] Connection header received
0806 09:12:54.127727 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:54.127735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:54.127877 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:54.128006 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:54.131526 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:54.133517 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:54.137809 (+ 4292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:54.138706 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:54.138871 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:54.144016 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:54.144021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:54.144024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:54.144111 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:54.148731 (+ 4620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:54.149645 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:54.149650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:54.149653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:54.149870 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:54.150600 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:54.150604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:54.150606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:54.151007 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:54.151241 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:54.152257 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0806 09:12:54.152507 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":283,"threads_started":1}
I20260806 09:12:54.154119 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:54.121124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:43989 (local address 127.3.69.2:33153)
0806 09:12:54.122630 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:54.122671 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:54.127371 (+ 4700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:54.128420 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:54.128448 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:54.129713 (+ 1265us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:12:54.131296 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:54.131327 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:54.133736 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:54.133746 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:54.137460 (+ 3714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:54.137523 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:54.139100 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:54.139111 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:54.139595 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:54.140524 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:12:54.140548 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:12:54.143756 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:12:54.148936 (+ 5180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:54.148953 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:54.148968 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:54.149495 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:54.149999 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:54.150004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:54.150008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:54.150442 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:54.151221 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:12:54.151230 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:12:54.151598 (+ 368us) client_negotiation.cc:769] Sending connection context
0806 09:12:54.152006 (+ 408us) client_negotiation.cc:240] Negotiation successful
0806 09:12:54.152661 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"thread_start_us":712,"threads_started":1}
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
I20260806 09:12:54.156191 19884 heartbeater.cc:347] Connected to a master server at 127.3.69.62:43989
I20260806 09:12:54.156600 19884 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
I20260806 09:12:54.157546 19884 heartbeater.cc:511] Master 127.3.69.62:43989 requested a full tablet report, sending...
I20260806 09:12:54.159314 19361 ts_manager.cc:194] Registered new tserver with Master: 8bafa8c243c44e879b47123521431965 (127.3.69.2:37603)
I20260806 09:12:54.160136 19361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:33153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:12:54.196303 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:40461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:43989
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:12:54.806265 19892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:12:54.806689 19892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:12:54.806993 19892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:12:54.818343 19892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:12:54.818652 19892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:12:54.818920 19892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:12:54.833627 19892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.69.20:40461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:43989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:12:54.835196 19892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:12:54.836633 19892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:12:54.856389 19898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:54.856314 19903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:12:54.856725 19899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:12:54.858263 19892 server_base.cc:1034] running on GCE node
I20260806 09:12:54.859303 19892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:12:54.860515 19892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:12:54.861848 19892 hybrid_clock.cc:648] HybridClock initialized: now 1786007574861717 us; error 55 us; skew 500 ppm
I20260806 09:12:54.862368 19892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:12:54 dist-test-slave-w4v5 krb5kdc[18661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007574, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:12:54.867722 19892 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:12:54.869663 19892 webserver.cc:460] Webserver started at http://127.3.69.3:40927/ using document root <none> and password file <none>
I20260806 09:12:54.870504 19892 fs_manager.cc:359] Metadata directory not provided
I20260806 09:12:54.870743 19892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:12:54.871275 19892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:12:54 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007574, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:12:54.889822 19892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "9bb694876bae42008a3d81f711f60ded"
format_stamp: "Formatted at 2026-08-06 09:12:54 on dist-test-slave-w4v5"
server_key: "d64d95dfba887af1ea3683eda290fd57"
server_key_iv: "c240da441f21a8c15ecca059dbb1d79e"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:54.891387 19892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "9bb694876bae42008a3d81f711f60ded"
format_stamp: "Formatted at 2026-08-06 09:12:54 on dist-test-slave-w4v5"
server_key: "d64d95dfba887af1ea3683eda290fd57"
server_key_iv: "c240da441f21a8c15ecca059dbb1d79e"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:54.900892 19892 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s
I20260806 09:12:54.920811 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:54.922566 19892 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260806 09:12:54.922801 19892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "9bb694876bae42008a3d81f711f60ded"
format_stamp: "Formatted at 2026-08-06 09:12:54 on dist-test-slave-w4v5"
server_key: "d64d95dfba887af1ea3683eda290fd57"
server_key_iv: "c240da441f21a8c15ecca059dbb1d79e"
server_key_version: "kuduclusterkey@0"
I20260806 09:12:54.923051 19892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:12:54.962316 19892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:12:54.968911 19892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:12:54.969412 19892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:12:54.971225 19892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:12:54.973760 19892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:12:54.973928 19892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:54.974172 19892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:12:54.974262 19892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:12:55.091758 19892 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:40919
I20260806 09:12:55.091935 20085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:40919 every 8 connection(s)
I20260806 09:12:55.093950 19892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:12:55.100775 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 19892
I20260806 09:12:55.101056 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:12:55.101902 3348 external_mini_cluster.cc:1428] Setting key fc67bff590a250dbc01ca9c788bad77d
Aug 06 09:12:55 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007574, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:12:55.134229 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.098905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:52365 (local address 127.3.69.62:43989)
0806 09:12:55.099990 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.099998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.104608 (+ 4610us) server_negotiation.cc:407] Connection header received
0806 09:12:55.109165 (+ 4557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.109172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.109356 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.109627 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:55.113735 (+ 4108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.115034 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.117386 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.118075 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.118224 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.125113 (+ 6889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:55.125122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:55.125127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:55.125173 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:55.129945 (+ 4772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.130831 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.130837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.130841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.131041 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.131554 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.131559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.131563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.132136 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:55.132328 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.133437 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.133761 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":558,"threads_started":1}
I20260806 09:12:55.135367 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.099805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:43989 (local address 127.3.69.3:52365)
0806 09:12:55.104246 (+ 4441us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:55.104321 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:55.108772 (+ 4451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:55.110393 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:55.110415 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:55.111670 (+ 1255us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:12:55.113465 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.113515 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.115315 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.115319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.117167 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.117180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.118449 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.118460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.118791 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.119888 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:12:55.119919 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:12:55.124780 (+ 4861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:12:55.130101 (+ 5321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:55.130111 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:55.130128 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:55.130711 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:55.131213 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:12:55.131218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:12:55.131221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:12:55.131423 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:12:55.132618 (+ 1195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:12:55.132627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:12:55.132999 (+ 372us) client_negotiation.cc:769] Sending connection context
0806 09:12:55.133338 (+ 339us) client_negotiation.cc:240] Negotiation successful
0806 09:12:55.133909 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3924,"thread_start_us":3386,"threads_started":1}
I20260806 09:12:55.137490 20086 heartbeater.cc:347] Connected to a master server at 127.3.69.62:43989
I20260806 09:12:55.137992 20086 heartbeater.cc:464] Registering TS with master...
I20260806 09:12:55.139271 20086 heartbeater.cc:511] Master 127.3.69.62:43989 requested a full tablet report, sending...
I20260806 09:12:55.141393 19361 ts_manager.cc:194] Registered new tserver with Master: 9bb694876bae42008a3d81f711f60ded (127.3.69.3:40919)
I20260806 09:12:55.142745 19361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:52365
I20260806 09:12:55.147468 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:12:55.164453 19884 heartbeater.cc:503] Master 127.3.69.62:43989 was elected leader, sending a full tablet report...
I20260806 09:12:55.171314 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.153338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.3.69.62:43989)
0806 09:12:55.153517 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.153525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.154457 (+ 932us) server_negotiation.cc:407] Connection header received
0806 09:12:55.154702 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.154708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.154861 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.154961 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:55.156525 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.157427 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.159206 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.160409 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.160578 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.163442 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:55.163447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:55.163451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:55.163485 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:55.167731 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.168580 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.168585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.168593 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.168778 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.169264 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.169268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.169270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.169583 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:55.169979 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.170406 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.170805 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:12:55.176193 19361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:12:55.187947 19361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:12:55.225584 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.206811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:37603 (local address 127.0.0.1:46540)
0806 09:12:55.208644 (+ 1833us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:55.208704 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:55.209167 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:55.211234 (+ 2067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:55.211245 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:55.211286 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:55.212440 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.212461 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.215270 (+ 2809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.215278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.219476 (+ 4198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.219496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.223541 (+ 4045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.223549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.223861 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.223919 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 09:12:55.224106 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 09:12:55.224396 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"thread_start_us":1083,"threads_started":1}
I20260806 09:12:55.227389 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.207003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.3.69.2:37603)
0806 09:12:55.208304 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.208314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.209457 (+ 1143us) server_negotiation.cc:407] Connection header received
0806 09:12:55.209587 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.209597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.209989 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.210656 (+ 667us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:55.212747 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.215017 (+ 2270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.219714 (+ 4697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.222323 (+ 2609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.222482 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.222982 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.225312 (+ 2330us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.225521 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":1495,"threads_started":1}
I20260806 09:12:55.233523 19779 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0684525b694743915022da51b36f35 (DEFAULT_TABLE table=test-table [id=14351b657a804dc986c5ed434fc70ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:12:55.235551 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.208234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.3.69.1:33295)
0806 09:12:55.223460 (+ 15226us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.223481 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.223533 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:12:55.223641 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.223650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.223993 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.224402 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:55.225837 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.227643 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.231858 (+ 4215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.233658 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.233828 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.234319 (+ 491us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.234609 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.234815 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14983,"thread_start_us":16914,"threads_started":1}
I20260806 09:12:55.235776 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.207581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:33295 (local address 127.0.0.1:59830)
0806 09:12:55.220875 (+ 13294us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:55.220904 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:55.221289 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:55.224742 (+ 3453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:55.224759 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:55.224788 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:55.225468 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.225486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.227815 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.227820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.231664 (+ 3844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.231681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.234088 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.234097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.234412 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.234455 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:12:55.234559 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 09:12:55.234706 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13031,"mutex_wait_us":23,"thread_start_us":13805,"threads_started":1}
I20260806 09:12:55.236100 19779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0684525b694743915022da51b36f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:12:55.242935 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.208933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:40919 (local address 127.0.0.1:35770)
0806 09:12:55.224024 (+ 15091us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:55.224049 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:55.224403 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:55.228668 (+ 4265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:55.228676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:55.228720 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:55.229888 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.229911 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.234537 (+ 4626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.234542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.238069 (+ 3527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.238086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.241541 (+ 3455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.241549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.241842 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.241885 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:12:55.242001 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 09:12:55.242148 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14850,"thread_start_us":14610,"threads_started":1}
I20260806 09:12:55.243381 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.210220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.3.69.3:40919)
0806 09:12:55.227582 (+ 17362us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.227592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.227654 (+ 62us) server_negotiation.cc:407] Connection header received
0806 09:12:55.227813 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.227822 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.228217 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.228677 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:55.230321 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.234285 (+ 3964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.239251 (+ 4966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.241129 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.241296 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.241777 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.242062 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.242313 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17089,"thread_start_us":16671,"threads_started":1}
I20260806 09:12:55.242687 19581 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0684525b694743915022da51b36f35 (DEFAULT_TABLE table=test-table [id=14351b657a804dc986c5ed434fc70ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:12:55.245339 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0684525b694743915022da51b36f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:12:55.247824 19986 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0684525b694743915022da51b36f35 (DEFAULT_TABLE table=test-table [id=14351b657a804dc986c5ed434fc70ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:12:55.249982 19986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0684525b694743915022da51b36f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:12:55.268996 20118 tablet_bootstrap.cc:492] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Bootstrap starting.
I20260806 09:12:55.270946 20116 tablet_bootstrap.cc:492] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Bootstrap starting.
I20260806 09:12:55.275808 20118 tablet_bootstrap.cc:654] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Neither blocks nor log segments found. Creating new log.
I20260806 09:12:55.277954 20116 tablet_bootstrap.cc:654] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Neither blocks nor log segments found. Creating new log.
I20260806 09:12:55.278451 20120 tablet_bootstrap.cc:492] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: Bootstrap starting.
I20260806 09:12:55.278985 20118 log.cc:824] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Log is configured to *not* fsync() on all Append() calls
I20260806 09:12:55.281039 20116 log.cc:824] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Log is configured to *not* fsync() on all Append() calls
I20260806 09:12:55.284586 20118 tablet_bootstrap.cc:492] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: No bootstrap required, opened a new log
I20260806 09:12:55.284999 20118 ts_tablet_manager.cc:1397] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260806 09:12:55.285666 20120 tablet_bootstrap.cc:654] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: Neither blocks nor log segments found. Creating new log.
I20260806 09:12:55.287175 20116 tablet_bootstrap.cc:492] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: No bootstrap required, opened a new log
I20260806 09:12:55.287588 20116 ts_tablet_manager.cc:1397] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260806 09:12:55.287689 20120 log.cc:824] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: Log is configured to *not* fsync() on all Append() calls
I20260806 09:12:55.295552 20120 tablet_bootstrap.cc:492] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: No bootstrap required, opened a new log
I20260806 09:12:55.295989 20120 ts_tablet_manager.cc:1397] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.016s
I20260806 09:12:55.304337 20118 raft_consensus.cc:348] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.305101 20118 raft_consensus.cc:374] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:12:55.305238 20118 raft_consensus.cc:729] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb694876bae42008a3d81f711f60ded, State: Initialized, Role: FOLLOWER
I20260806 09:12:55.304983 20116 raft_consensus.cc:348] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.305696 20116 raft_consensus.cc:374] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:12:55.305845 20116 raft_consensus.cc:729] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bafa8c243c44e879b47123521431965, State: Initialized, Role: FOLLOWER
I20260806 09:12:55.306288 20118 consensus_queue.cc:260] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.306790 20116 consensus_queue.cc:260] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.308903 20086 heartbeater.cc:503] Master 127.3.69.62:43989 was elected leader, sending a full tablet report...
I20260806 09:12:55.309458 20120 raft_consensus.cc:348] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.310140 20120 raft_consensus.cc:374] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:12:55.310326 20120 raft_consensus.cc:729] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc590cb05c984cd38f19175744f406fc, State: Initialized, Role: FOLLOWER
I20260806 09:12:55.311183 20120 consensus_queue.cc:260] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.312465 20118 ts_tablet_manager.cc:1428] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Time spent starting tablet: real 0.027s user 0.018s sys 0.006s
I20260806 09:12:55.316220 20116 ts_tablet_manager.cc:1428] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Time spent starting tablet: real 0.028s user 0.022s sys 0.004s
I20260806 09:12:55.320019 20120 ts_tablet_manager.cc:1428] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: Time spent starting tablet: real 0.024s user 0.008s sys 0.016s
W20260806 09:12:55.347601 20087 tablet.cc:2369] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:12:55.370600 19885 tablet.cc:2369] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:12:55.385666 20126 raft_consensus.cc:482] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:12:55.386302 20126 raft_consensus.cc:504] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.388978 20126 leader_election.cc:288] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bafa8c243c44e879b47123521431965 (127.3.69.2:37603), 9bb694876bae42008a3d81f711f60ded (127.3.69.3:40919)
I20260806 09:12:55.400544 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.389774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:37603 (local address 127.3.69.1:43831)
0806 09:12:55.391490 (+ 1716us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:55.391512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:55.391757 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:55.392365 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:55.392368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:55.392390 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:55.393428 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.393442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.395140 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.395145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.397402 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.397414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.399455 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.399459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.399708 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.399792 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 09:12:55.399885 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 09:12:55.400034 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":865,"threads_started":1}
I20260806 09:12:55.400545 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.389477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:43831 (local address 127.3.69.2:37603)
0806 09:12:55.389770 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.389775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.391634 (+ 1859us) server_negotiation.cc:407] Connection header received
0806 09:12:55.391991 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.391997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.392071 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.392124 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:55.393590 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.394947 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.397561 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.399113 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.399305 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.399460 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.399957 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.400157 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:12:55.402158 20130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.391394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:40919 (local address 127.3.69.1:58211)
0806 09:12:55.393437 (+ 2043us) negotiation.cc:107] Waiting for socket to connect
0806 09:12:55.393454 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:12:55.393589 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:12:55.393991 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:12:55.393994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:12:55.394014 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:12:55.394946 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.394958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.396517 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.396525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.399017 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:12:55.399030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.401295 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.401319 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:12:55.401543 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.401569 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 09:12:55.401647 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 09:12:55.401743 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1889,"thread_start_us":1508,"threads_started":1}
I20260806 09:12:55.402323 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.391589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:58211 (local address 127.3.69.3:40919)
0806 09:12:55.391899 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.391906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.393615 (+ 1709us) server_negotiation.cc:407] Connection header received
0806 09:12:55.393722 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.393727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.393842 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.393972 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:12:55.395185 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.396295 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.399241 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.400991 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.401091 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.401164 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.401756 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.401918 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 09:12:55.402087 19809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0684525b694743915022da51b36f35" candidate_uuid: "cc590cb05c984cd38f19175744f406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bafa8c243c44e879b47123521431965" is_pre_election: true
I20260806 09:12:55.403285 19809 raft_consensus.cc:2454] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc590cb05c984cd38f19175744f406fc in term 0.
I20260806 09:12:55.403695 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0684525b694743915022da51b36f35" candidate_uuid: "cc590cb05c984cd38f19175744f406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb694876bae42008a3d81f711f60ded" is_pre_election: true
I20260806 09:12:55.404387 19517 leader_election.cc:302] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bafa8c243c44e879b47123521431965, cc590cb05c984cd38f19175744f406fc; no voters:
I20260806 09:12:55.404578 20013 raft_consensus.cc:2454] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc590cb05c984cd38f19175744f406fc in term 0.
I20260806 09:12:55.404983 20126 raft_consensus.cc:2781] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:12:55.405262 20126 raft_consensus.cc:482] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:12:55.405498 20126 raft_consensus.cc:3037] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:12:55.409475 20126 raft_consensus.cc:504] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.410466 20126 leader_election.cc:288] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [CANDIDATE]: Term 1 election: Requested vote from peers 8bafa8c243c44e879b47123521431965 (127.3.69.2:37603), 9bb694876bae42008a3d81f711f60ded (127.3.69.3:40919)
I20260806 09:12:55.410987 19809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0684525b694743915022da51b36f35" candidate_uuid: "cc590cb05c984cd38f19175744f406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bafa8c243c44e879b47123521431965"
I20260806 09:12:55.411209 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0684525b694743915022da51b36f35" candidate_uuid: "cc590cb05c984cd38f19175744f406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb694876bae42008a3d81f711f60ded"
I20260806 09:12:55.411502 20013 raft_consensus.cc:3037] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:12:55.411433 19809 raft_consensus.cc:3037] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:12:55.415751 20013 raft_consensus.cc:2454] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc590cb05c984cd38f19175744f406fc in term 1.
I20260806 09:12:55.416033 19809 raft_consensus.cc:2454] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc590cb05c984cd38f19175744f406fc in term 1.
I20260806 09:12:55.416358 19517 leader_election.cc:302] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bb694876bae42008a3d81f711f60ded, cc590cb05c984cd38f19175744f406fc; no voters:
I20260806 09:12:55.416983 20126 raft_consensus.cc:2781] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:12:55.417953 20126 raft_consensus.cc:686] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 1 LEADER]: Becoming Leader. State: Replica: cc590cb05c984cd38f19175744f406fc, State: Running, Role: LEADER
I20260806 09:12:55.418634 20126 consensus_queue.cc:237] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } }
I20260806 09:12:55.425122 19361 catalog_manager.cc:5495] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc reported cstate change: term changed from 0 to 1, leader changed from <none> to cc590cb05c984cd38f19175744f406fc (127.3.69.1). New cstate: current_term: 1 leader_uuid: "cc590cb05c984cd38f19175744f406fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc590cb05c984cd38f19175744f406fc" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 33295 } health_report { overall_health: HEALTHY } } }
I20260806 09:12:55.485028 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.477765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.3.69.1:33295)
0806 09:12:55.478039 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.478044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.478062 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:12:55.478157 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.478160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.478261 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.478396 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:12:55.479011 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.480234 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.482440 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.483035 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.483229 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.483895 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:12:55.484442 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:12:55.484560 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.484656 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.484726 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:12:55.495942 19809 raft_consensus.cc:1261] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 1 FOLLOWER]: Refusing update from remote peer cc590cb05c984cd38f19175744f406fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:12:55.496095 20013 raft_consensus.cc:1261] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 1 FOLLOWER]: Refusing update from remote peer cc590cb05c984cd38f19175744f406fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:12:55.497048 20131 consensus_queue.cc:1035] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bafa8c243c44e879b47123521431965" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 37603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:12:55.497671 20126 consensus_queue.cc:1035] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb694876bae42008a3d81f711f60ded" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:12:55.513729 20134 mvcc.cc:205] Tried to move back new op lower bound from 7315487029218488320 to 7315487028920111104. Current Snapshot: MvccSnapshot[applied={T|T < 7315487029218488320}]
I20260806 09:12:55.537876 19361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:12:55.538290 19361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:12:55.543385 19361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 709224bd6b9d4641900b0cd7e79f50cd: Sending DeleteTablet for 3 replicas of tablet 9f0684525b694743915022da51b36f35
I20260806 09:12:55.544831 19986 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0684525b694743915022da51b36f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:12:55 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:35770
I20260806 09:12:55.545257 19779 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0684525b694743915022da51b36f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:12:55 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:46540
I20260806 09:12:55.545849 19581 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0684525b694743915022da51b36f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:12:55 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:59830
I20260806 09:12:55.547158 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 19475
I20260806 09:12:55.562654 20150 tablet_replica.cc:330] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: stopping tablet replica
I20260806 09:12:55.563407 20150 raft_consensus.cc:2229] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:12:55.564473 20150 raft_consensus.cc:2258] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:12:55.566659 20151 tablet_replica.cc:330] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: stopping tablet replica
I20260806 09:12:55.567384 20151 raft_consensus.cc:2229] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:12:55.568050 20150 ts_tablet_manager.cc:1905] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:12:55.568608 20151 raft_consensus.cc:2258] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:12:55.569378 20147 tablet_replica.cc:330] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: stopping tablet replica
I20260806 09:12:55.569929 20147 raft_consensus.cc:2229] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:12:55.570869 20147 raft_consensus.cc:2258] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:12:55.571470 20151 ts_tablet_manager.cc:1905] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 09:12:55 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:12:55.580715 20147 ts_tablet_manager.cc:1905] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:12:55.582182 20150 ts_tablet_manager.cc:1918] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:12:55.582489 20150 log.cc:1196] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/9f0684525b694743915022da51b36f35
I20260806 09:12:55.583212 20151 ts_tablet_manager.cc:1918] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:12:55.583472 20150 ts_tablet_manager.cc:1939] T 9f0684525b694743915022da51b36f35 P 8bafa8c243c44e879b47123521431965: Deleting consensus metadata
I20260806 09:12:55.583606 20151 log.cc:1196] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/9f0684525b694743915022da51b36f35
I20260806 09:12:55.584563 20151 ts_tablet_manager.cc:1939] T 9f0684525b694743915022da51b36f35 P cc590cb05c984cd38f19175744f406fc: Deleting consensus metadata
I20260806 09:12:55.586210 19342 catalog_manager.cc:4841] TS 8bafa8c243c44e879b47123521431965 (127.3.69.2:37603): tablet 9f0684525b694743915022da51b36f35 (table test-table [id=14351b657a804dc986c5ed434fc70ad2]) successfully deleted
I20260806 09:12:55.587920 19337 catalog_manager.cc:4841] TS cc590cb05c984cd38f19175744f406fc (127.3.69.1:33295): tablet 9f0684525b694743915022da51b36f35 (table test-table [id=14351b657a804dc986c5ed434fc70ad2]) successfully deleted
I20260806 09:12:55.591651 20147 ts_tablet_manager.cc:1918] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:12:55.591931 20147 log.cc:1196] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/9f0684525b694743915022da51b36f35
I20260806 09:12:55.592653 20147 ts_tablet_manager.cc:1939] T 9f0684525b694743915022da51b36f35 P 9bb694876bae42008a3d81f711f60ded: Deleting consensus metadata
I20260806 09:12:55.594949 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.549160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.3.69.1:33295)
0806 09:12:55.549360 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.549366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.559973 (+ 10607us) server_negotiation.cc:407] Connection header received
0806 09:12:55.560671 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.560678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.560793 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.560902 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:55.563802 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.565209 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.568054 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.569293 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.569493 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.581933 (+ 12440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:55.581954 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:55.581974 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:55.582041 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:55.589597 (+ 7556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.590939 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.590949 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.590953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.591322 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.592144 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.592153 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.592159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.592876 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:55.593519 (+ 643us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.593762 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.594348 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 09:12:55.595106 19342 catalog_manager.cc:4841] TS 9bb694876bae42008a3d81f711f60ded (127.3.69.3:40919): tablet 9f0684525b694743915022da51b36f35 (table test-table [id=14351b657a804dc986c5ed434fc70ad2]) successfully deleted
I20260806 09:12:55.595997 19669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:12:55.701195 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 19475
I20260806 09:12:55.724416 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 19683
Aug 06 09:12:55 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:12:55.741032 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.725282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.3.69.2:37603)
0806 09:12:55.725484 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.725489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.725508 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:12:55.725565 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.725569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.725633 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.725722 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:55.727019 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.727974 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.729446 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.730092 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.730197 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.734741 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:55.734754 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:55.734770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:55.734818 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:55.738369 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.738986 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.738993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.738996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.739189 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.739593 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.739596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.739598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.739937 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:55.740174 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.740425 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.740645 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:12:55.741783 19877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:12:55.835268 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 19683
I20260806 09:12:55.864902 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 19892
Aug 06 09:12:55 dist-test-slave-w4v5 krb5kdc[18661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:12:55.880545 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:12:55.865820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.3.69.3:40919)
0806 09:12:55.866012 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:12:55.866017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:12:55.866038 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:12:55.866176 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:12:55.866180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:12:55.866272 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:12:55.866359 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:12:55.867697 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.868419 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.869631 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:12:55.870003 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:12:55.870095 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:12:55.874633 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:12:55.874649 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:12:55.874666 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:12:55.874702 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:12:55.877458 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.878154 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.878162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.878164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.878358 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:12:55.878774 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:12:55.878780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:12:55.878783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:12:55.879288 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:12:55.879610 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 09:12:55.879939 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 09:12:55.880164 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:12:55.881198 20078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:12:55.974535 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 19892
I20260806 09:12:56.000661 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 19319
I20260806 09:12:56.001981 19409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:12:56.117736 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 19319
I20260806 09:12:56.152352 3348 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 09:12:56.497689 3348 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 09:12:56.497983 3348 mini_ranger.cc:66] Stopping Ranger...
I20260806 09:12:56.843523 3348 mini_ranger.cc:68] Stopped Ranger
2026-08-06 09:12:56.843 UTC [18717] LOG: received smart shutdown request
2026-08-06 09:12:56.847 UTC [18717] LOG: background worker "logical replication launcher" (PID 18732) exited with exit code 1
2026-08-06 09:12:56.848 UTC [18721] LOG: shutting down
2026-08-06 09:12:57.032 UTC [18717] LOG: database system is shut down
2026-08-06T09:12:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53012 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:12:57 dist-test-slave-w4v5 krb5kdc[20168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:12:57 dist-test-slave-w4v5 krb5kdc[20168](info): set up 2 sockets
Aug 06 09:12:57 dist-test-slave-w4v5 krb5kdc[20168](info): commencing operation
krb5kdc: starting...
W20260806 09:12:58.497478 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:12:58 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:12:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.69.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.69.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/rangeradmin_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/rangeradmin_127.3.69.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.69.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.69.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/rangerlookup_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/rangerlookup_127.3.69.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.69.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.69.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/HTTP_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/HTTP_127.3.69.20@KRBTEST.COM.keytab.
I20260806 09:12:58.862275 3348 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6yYR2E/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/postgres -l logfile start
2026-08-06 09:13:01.423 UTC [20215] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 09:13:01.423 UTC [20215] LOG: listening on IPv4 address "127.3.69.20", port 41197
2026-08-06 09:13:01.426 UTC [20215] LOG: listening on Unix socket "/tmp/.s.PGSQL.41197"
2026-08-06 09:13:01.458 UTC [20218] LOG: database system was shut down at 2026-08-06 09:12:59 UTC
2026-08-06 09:13:01.461 UTC [20215] LOG: database system is ready to accept connections
I20260806 09:13:02.625068 3348 mini_postgres.cc:96] Postgres bound to 41197
2026-08-06 09:13:02.647 UTC [20228] FATAL: database "slave" does not exist
127.3.69.20:41197 - accepting connections
I20260806 09:13:02.648114 3348 mini_ranger.cc:161] Starting Ranger...
I20260806 09:13:02.678959 3348 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 09:13:03.160418 3348 mini_ranger.cc:87] Created ranger Postgres database
I20260806 09:13:03.160521 3348 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-admin
2026-08-06 09:13:03,460 [I] DB FLAVOR :POSTGRES
2026-08-06 09:13:03,461 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:13:03,461 [I] Checking connection..
2026-08-06 09:13:03,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 09:13:03,723 [I] Checking connection passed.
2026-08-06 09:13:03,723 [I] --------- Verifying version history table ---------
2026-08-06 09:13:03,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:13:03,979 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:13:03,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:13:04,230 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:13:04,230 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 09:13:04,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 09:13:04.482 UTC [20322] WARNING: there is no transaction in progress
2026-08-06 09:13:04,491 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 09:13:04,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:13:04,760 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 09:13:04,760 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:13:04,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 09:13:05,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 09:13:05,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w4v5.c.gcp-upstream.internal','N') ;"
2026-08-06 09:13:05,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:13:05,805 [I] Table x_portal_user does not exist in database ranger
2026-08-06 09:13:05,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:13:06,053 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 09:13:06,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:13:06,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:13:06,568 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 09:13:06,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 09:13:07.580 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.598 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.615 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.631 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.646 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.667 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.722 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.729 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.743 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.753 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.763 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.777 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.788 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.795 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.802 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:07.809 UTC [20467] WARNING: there is no transaction in progress
2026-08-06 09:13:08,475 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 09:13:08,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 09:13:08,737 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 09:13:08,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:13:09,013 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 09:13:09,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:13:09,254 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 09:13:09,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:13:09,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:13:09,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w4v5.c.gcp-upstream.internal';"
2026-08-06 09:13:10,031 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 09:13:10,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 09:13:10,275 [I] DB_PATCHES have already been applied
I20260806 09:13:10.282330 3348 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-admin:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6yYR2E/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 09:13:10.282429 3348 mini_ranger.cc:193] Using host: 127.3.69.20
I20260806 09:13:10.302623 3348 mini_ranger.cc:227] Ranger admin URL: http://127.3.69.20:47563
Aug 06, 2026 9:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 9:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 9:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 9:13:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 9:13:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 09:13:11.698359 3348 mini_ranger.cc:160] Time spent starting Ranger: real 9.050s user 0.003s sys 0.051s
Aug 06, 2026 9:13:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 9:13:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 06, 2026 9:13:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 9:13:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 9:13:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 09:13:33.064836 3348 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.69.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.69.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/rangerkms_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.69.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/rangerkms_127.3.69.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.69.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.69.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.69.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/HTTP_127.3.69.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.69.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/HTTP_127.3.69.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 09:13:33 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007613, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 09:13:33.296178 3348 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 09:13:33.326331 3348 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 09:13:34.137188 3348 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 09:13:34.137336 3348 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms
I20260806 09:13:34.137383 3348 mini_ranger_kms.cc:227] Using postgres at 127.3.69.20:41197
2026-08-06 09:13:34,581 [I] DB FLAVOR :POSTGRES
2026-08-06 09:13:34,581 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:13:34,581 [I] Checking connection
2026-08-06 09:13:34,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 09:13:34,833 [I] connection success
2026-08-06 09:13:34,834 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 09:13:34,834 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 09:13:34,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 09:13:35,073 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 09:13:35,074 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:13:35,074 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 09:13:35,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.69.20:41197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 09:13:35,344 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 09:13:35.821254 3348 mini_ranger_kms.cc:310] Created kms service
I20260806 09:13:35.998467 3348 mini_ranger_kms.cc:326] Created kudu user
I20260806 09:13:36.045925 3348 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 09:13:36 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007613, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:36.457837 3348 mini_ranger_kms.cc:384] Added ranger policy
I20260806 09:13:36.458182 3348 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms:/tmp/dist-test-task6yYR2E/build/asan/bin/postgresql.jar:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6yYR2E/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 09:13:36.458240 3348 mini_ranger_kms.cc:240] Using host: 127.3.69.20
I20260806 09:13:36.481427 3348 mini_ranger_kms.cc:276] Ranger KMS PID: 20720
I20260806 09:13:36.481559 3348 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.69.20:58635
Aug 06, 2026 9:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 9:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 9:13:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 9:13:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 09:13:39 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.69.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:39.234371 3348 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.938s user 0.074s sys 0.403s
I20260806 09:13:39.234649 3348 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 09:13:39.234854 3348 mini_ranger_kms.cc:401] 127.3.69.20:58635/kms/v1/keys
Aug 06 09:13:39 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.69.20@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
Aug 06 09:13:39 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.69.20@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
Aug 06 09:13:42 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:13:42.152451 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36197
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:36197
--ranger_config_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:58635/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:13:42.504658 20786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:42.504956 20786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:42.505048 20786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:42.512364 20786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:13:42.512480 20786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:42.512524 20786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:42.512574 20786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:13:42.512632 20786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:13:42.522573 20786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.69.20:58635/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:36197
--ranger_config_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:42.523803 20786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:42.525086 20786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:42.537843 20794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:42.537787 20800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:42.537822 20795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:42.538353 20786 server_base.cc:1034] running on GCE node
I20260806 09:13:42.539383 20786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:42.540803 20786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:42.542061 20786 hybrid_clock.cc:648] HybridClock initialized: now 1786007622542005 us; error 44 us; skew 500 ppm
I20260806 09:13:42.542443 20786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:42 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:42.548681 20786 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:13:42.550212 20786 webserver.cc:460] Webserver started at http://127.3.69.62:43633/ using document root <none> and password file <none>
I20260806 09:13:42.550861 20786 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:42.550958 20786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:13:42.551306 20786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:13:42 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:42.617442 20786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "e794ec9bf3904b8eb769204b60660c16"
format_stamp: "Formatted at 2026-08-06 09:13:42 on dist-test-slave-w4v5"
server_key: "0095fc3d6e4dc46a3e8b31aa9130e7a0"
server_key_iv: "6bd2b462cef6b5874ab1284cc979cbe9"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:42.618326 20786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "e794ec9bf3904b8eb769204b60660c16"
format_stamp: "Formatted at 2026-08-06 09:13:42 on dist-test-slave-w4v5"
server_key: "0095fc3d6e4dc46a3e8b31aa9130e7a0"
server_key_iv: "6bd2b462cef6b5874ab1284cc979cbe9"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:42.624053 20786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 09:13:42.640833 20807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:42.642386 20786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:13:42.642597 20786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "e794ec9bf3904b8eb769204b60660c16"
format_stamp: "Formatted at 2026-08-06 09:13:42 on dist-test-slave-w4v5"
server_key: "0095fc3d6e4dc46a3e8b31aa9130e7a0"
server_key_iv: "6bd2b462cef6b5874ab1284cc979cbe9"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:42.642825 20786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:42.676396 20786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:42.680147 20786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:42.680501 20786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:42.699591 20786 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:36197
I20260806 09:13:42.699636 20890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:36197 every 8 connection(s)
I20260806 09:13:42.700989 20786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:13:42.704442 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 20786
I20260806 09:13:42.704666 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:13:42.704914 20892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:13:42.705464 3348 external_mini_cluster.cc:1428] Setting key 2abfd6174467ee4014a11b80bb1acd8a
I20260806 09:13:42.719609 20892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Bootstrap starting.
Aug 06 09:13:42 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:42.723632 20892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Neither blocks nor log segments found. Creating new log.
I20260806 09:13:42.725035 20892 log.cc:824] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Log is configured to *not* fsync() on all Append() calls
I20260806 09:13:42.728682 20892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: No bootstrap required, opened a new log
I20260806 09:13:42.730234 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:42.707468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.3.69.62:36197)
0806 09:13:42.708379 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:42.708395 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:42.708450 (+ 55us) server_negotiation.cc:407] Connection header received
0806 09:13:42.710197 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:42.710237 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:42.710736 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:42.711196 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:42.712428 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:42.714414 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:42.715971 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:42.717074 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:42.717283 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:42.721555 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:42.721568 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:42.721588 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:42.721632 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:42.725534 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:42.726518 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:42.726524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:42.726531 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:42.726762 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:42.727159 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:42.727164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:42.727167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:42.727860 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:42.728170 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:42.728522 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 09:13:42.728910 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":392,"threads_started":1}
I20260806 09:13:42.737401 20892 raft_consensus.cc:348] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } }
I20260806 09:13:42.737735 20892 raft_consensus.cc:374] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:13:42.737809 20892 raft_consensus.cc:729] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e794ec9bf3904b8eb769204b60660c16, State: Initialized, Role: FOLLOWER
I20260806 09:13:42.738346 20892 consensus_queue.cc:260] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } }
I20260806 09:13:42.738582 20892 raft_consensus.cc:388] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:13:42.738687 20892 raft_consensus.cc:482] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:13:42.738823 20892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:13:42.740932 20892 raft_consensus.cc:504] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } }
I20260806 09:13:42.741489 20892 leader_election.cc:302] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e794ec9bf3904b8eb769204b60660c16; no voters:
I20260806 09:13:42.742028 20892 leader_election.cc:288] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:13:42.742187 20902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:13:42.743029 20902 raft_consensus.cc:686] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 1 LEADER]: Becoming Leader. State: Replica: e794ec9bf3904b8eb769204b60660c16, State: Running, Role: LEADER
I20260806 09:13:42.743587 20902 consensus_queue.cc:237] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } }
I20260806 09:13:42.744227 20892 sys_catalog.cc:564] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:13:42.746246 20904 sys_catalog.cc:455] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e794ec9bf3904b8eb769204b60660c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } } }
I20260806 09:13:42.746752 20904 sys_catalog.cc:458] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: This master's current role is: LEADER
I20260806 09:13:42.746685 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e794ec9bf3904b8eb769204b60660c16. Latest consensus state: current_term: 1 leader_uuid: "e794ec9bf3904b8eb769204b60660c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } } }
I20260806 09:13:42.746958 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: This master's current role is: LEADER
I20260806 09:13:42.749274 20912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:13:42.754675 20912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:13:42.755482 20892 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 09:13:42.776989 20912 catalog_manager.cc:1347] Generated new cluster ID: fbba69464fd34e6cad7136c50ecde290
I20260806 09:13:42.777163 20912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:13:42.797627 20912 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:13:42.799791 20912 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:13:42.818490 20912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Generated new TSK 0
I20260806 09:13:42.820120 20912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:13:43.582741 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:58635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36197
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:13:43 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 09:13:43.988283 20957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:43.988662 20957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:43.988852 20957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:43.997814 20957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:43.998008 20957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:43.998241 20957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:13:44.009816 20957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.69.20:58635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:44.011328 20957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:44.012756 20957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:44.032959 20973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:44.033835 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:44.035876 20957 server_base.cc:1034] running on GCE node
W20260806 09:13:44.036477 20981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:44.037379 20957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:44.038740 20957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:44.040035 20957 hybrid_clock.cc:648] HybridClock initialized: now 1786007624039902 us; error 115 us; skew 500 ppm
I20260806 09:13:44.040444 20957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:44 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:44.045049 20957 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:13:44.052767 20957 webserver.cc:460] Webserver started at http://127.3.69.1:36657/ using document root <none> and password file <none>
I20260806 09:13:44.053812 20957 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:44.053960 20957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:13:44.054342 20957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:13:44 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:44.076908 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "8c6df30184d2490cbf4a5889b932b0f9"
format_stamp: "Formatted at 2026-08-06 09:13:44 on dist-test-slave-w4v5"
server_key: "3cde8514e1b6d3f95441e32f08589858"
server_key_iv: "623534543b504383b53b072df2b25a8f"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:44.078164 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "8c6df30184d2490cbf4a5889b932b0f9"
format_stamp: "Formatted at 2026-08-06 09:13:44 on dist-test-slave-w4v5"
server_key: "3cde8514e1b6d3f95441e32f08589858"
server_key_iv: "623534543b504383b53b072df2b25a8f"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:44.103929 20957 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.012s sys 0.008s
I20260806 09:13:44.127532 20991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:44.129361 20957 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.004s
I20260806 09:13:44.129577 20957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "8c6df30184d2490cbf4a5889b932b0f9"
format_stamp: "Formatted at 2026-08-06 09:13:44 on dist-test-slave-w4v5"
server_key: "3cde8514e1b6d3f95441e32f08589858"
server_key_iv: "623534543b504383b53b072df2b25a8f"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:44.129860 20957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:44.151757 20957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:44.157886 20957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:44.158291 20957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:44.159724 20957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:13:44.161607 20957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:13:44.161723 20957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:44.161855 20957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:13:44.161909 20957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:13:44 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:44.697078 21162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:42027 every 8 connection(s)
I20260806 09:13:44.697418 20957 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:42027
I20260806 09:13:44.699299 20957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:13:44.704084 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 20957
I20260806 09:13:44.704331 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:13:44.705343 3348 external_mini_cluster.cc:1428] Setting key 16f4af3ecb9cf9d37e6bc9052272b272
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Aug 06 09:13:44 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
Aug 06 09:13:44 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:44.784822 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:44.709467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36197 (local address 127.3.69.1:38059)
0806 09:13:44.727371 (+ 17904us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:44.727442 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:44.735207 (+ 7765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:44.747621 (+ 12414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:44.747635 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:44.748379 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:44.749309 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:44.749339 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:44.754348 (+ 5009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:44.754354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:44.756733 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:44.756847 (+ 114us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:44.759720 (+ 2873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:44.759731 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:44.760036 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:44.761014 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:44.761049 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:44.765233 (+ 4184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:44.775323 (+ 10090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:44.775338 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:44.775361 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:44.776227 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:44.776771 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:44.776779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:44.776785 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:44.777063 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:44.781574 (+ 4511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:44.781586 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:44.782124 (+ 538us) client_negotiation.cc:769] Sending connection context
0806 09:13:44.782475 (+ 351us) client_negotiation.cc:240] Negotiation successful
0806 09:13:44.783182 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17478,"thread_start_us":20347,"threads_started":1}
I20260806 09:13:44.787158 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:44.708620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:38059 (local address 127.3.69.62:36197)
0806 09:13:44.746343 (+ 37723us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:44.746349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:44.746374 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:13:44.746443 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:44.746448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:44.746542 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:44.751277 (+ 4735us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:44.751509 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:44.752863 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:44.757353 (+ 4490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:44.758502 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:44.765519 (+ 7017us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:44.765832 (+ 313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:44.765840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:44.765848 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:44.765892 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:44.772249 (+ 6357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:44.776433 (+ 4184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:44.776441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:44.776447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:44.776626 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:44.777229 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:44.777236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:44.777242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:44.777629 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:44.786119 (+ 8490us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:44.786333 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 09:13:44.786625 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37576,"thread_start_us":38039,"threads_started":1}
I20260806 09:13:44.795885 21164 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36197
I20260806 09:13:44.796278 21164 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:44.797298 21164 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
I20260806 09:13:44.800951 20837 ts_manager.cc:194] Registered new tserver with Master: 8c6df30184d2490cbf4a5889b932b0f9 (127.3.69.1:42027)
I20260806 09:13:44.803022 20837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:38059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
W20260806 09:13:44.816079 3348 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.69.2: real 0.105s user 0.000s sys 0.024s
I20260806 09:13:44.817081 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:58635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36197
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:13:45.257723 21179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:45.258074 21179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:45.258250 21179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:45.265759 21179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:45.265949 21179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:45.266129 21179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:13:45.276847 21179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.69.20:58635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:45.278349 21179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:45.279742 21179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:45.295960 21197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:45.296033 21198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:45.297197 21201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:45.297319 21179 server_base.cc:1034] running on GCE node
I20260806 09:13:45.298148 21179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:45.299285 21179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:45.300561 21179 hybrid_clock.cc:648] HybridClock initialized: now 1786007625300487 us; error 68 us; skew 500 ppm
I20260806 09:13:45.300959 21179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:45 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:45.306852 21179 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:13:45.308670 21179 webserver.cc:460] Webserver started at http://127.3.69.2:34613/ using document root <none> and password file <none>
I20260806 09:13:45.309409 21179 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:45.309544 21179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:13:45.309893 21179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:13:45 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:45.326782 21179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "ce6c18bfeabd4fbdbb73a956c1705d19"
format_stamp: "Formatted at 2026-08-06 09:13:45 on dist-test-slave-w4v5"
server_key: "be268c1dc69d36d1e9dbb33b9d14680c"
server_key_iv: "70723c74f7cd6e4cd89f501d80dc2960"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:45.327958 21179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "ce6c18bfeabd4fbdbb73a956c1705d19"
format_stamp: "Formatted at 2026-08-06 09:13:45 on dist-test-slave-w4v5"
server_key: "be268c1dc69d36d1e9dbb33b9d14680c"
server_key_iv: "70723c74f7cd6e4cd89f501d80dc2960"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:45.335424 21179 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260806 09:13:45.353358 21212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:45.355326 21179 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:13:45.355525 21179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "ce6c18bfeabd4fbdbb73a956c1705d19"
format_stamp: "Formatted at 2026-08-06 09:13:45 on dist-test-slave-w4v5"
server_key: "be268c1dc69d36d1e9dbb33b9d14680c"
server_key_iv: "70723c74f7cd6e4cd89f501d80dc2960"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:45.355784 21179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:45.383888 21179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:45.389372 21179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:45.389719 21179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:45.390985 21179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:13:45.392846 21179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:13:45.392942 21179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:45.393134 21179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:13:45.393229 21179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:45.475345 21179 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:33813
I20260806 09:13:45.475366 21377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:33813 every 8 connection(s)
I20260806 09:13:45.476779 21179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:13:45.481894 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 21179
I20260806 09:13:45.482079 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:13:45.482595 3348 external_mini_cluster.cc:1428] Setting key 940ca637ecb71cfbc3f19911b73e4226
Aug 06 09:13:45 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:45.505614 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:45.480491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:34013 (local address 127.3.69.62:36197)
0806 09:13:45.481115 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:45.481122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:45.482162 (+ 1040us) server_negotiation.cc:407] Connection header received
0806 09:13:45.484599 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:45.484603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:45.484696 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:45.484806 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:45.486968 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:45.488243 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:45.491932 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:45.492651 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:45.492767 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:45.497707 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:45.497712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:45.497716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:45.497756 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:45.501383 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:45.502443 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:45.502451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:45.502457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:45.502631 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:45.503209 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:45.503217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:45.503223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:45.503632 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:45.503774 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:45.504819 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0806 09:13:45.505091 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":312,"threads_started":1}
I20260806 09:13:45.508508 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:45.481077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36197 (local address 127.3.69.2:34013)
0806 09:13:45.481976 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:45.482024 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:45.484299 (+ 2275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:45.485251 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:45.485263 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:45.485891 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:45.486757 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:45.486786 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:45.489290 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:45.489298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:45.491750 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:45.491769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:45.493123 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:45.493133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:45.493467 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:45.494297 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:45.494333 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:45.497449 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:45.501597 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:45.501619 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:45.501639 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:45.502303 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:45.502780 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:45.502787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:45.502793 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:45.503033 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:45.503877 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:45.503888 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:45.504327 (+ 439us) client_negotiation.cc:769] Sending connection context
0806 09:13:45.506290 (+ 1963us) client_negotiation.cc:240] Negotiation successful
0806 09:13:45.506886 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":1946,"threads_started":1}
I20260806 09:13:45.510412 21379 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36197
I20260806 09:13:45.510821 21379 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:45.511709 21379 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
I20260806 09:13:45.513500 20837 ts_manager.cc:194] Registered new tserver with Master: ce6c18bfeabd4fbdbb73a956c1705d19 (127.3.69.2:33813)
I20260806 09:13:45.514832 20837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:34013
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:13:45.556746 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:58635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36197
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:13:45.807237 21164 heartbeater.cc:503] Master 127.3.69.62:36197 was elected leader, sending a full tablet report...
W20260806 09:13:45.904934 21388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:45.905267 21388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:45.905445 21388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:45.912599 21388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:45.912757 21388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:45.912945 21388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:13:45.922391 21388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.69.20:58635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:45.923770 21388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:45.925053 21388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:45.938269 21400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:45.938314 21403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:45.938736 21398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:45.939217 21388 server_base.cc:1034] running on GCE node
I20260806 09:13:45.939810 21388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:45.940743 21388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:45.941975 21388 hybrid_clock.cc:648] HybridClock initialized: now 1786007625941941 us; error 49 us; skew 500 ppm
I20260806 09:13:45.942287 21388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:45 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:45.945565 21388 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:13:45.947360 21388 webserver.cc:460] Webserver started at http://127.3.69.3:45261/ using document root <none> and password file <none>
I20260806 09:13:45.948138 21388 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:45.948325 21388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:13:45.948670 21388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:13:45 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:45.963825 21388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "a4705b50b1e44cbcb975fa05fb10b35e"
format_stamp: "Formatted at 2026-08-06 09:13:45 on dist-test-slave-w4v5"
server_key: "24d3ebb873b11b1efab2f4851a4f3a57"
server_key_iv: "1358d968348eec80b79ad43cb5458b64"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:45.964717 21388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "a4705b50b1e44cbcb975fa05fb10b35e"
format_stamp: "Formatted at 2026-08-06 09:13:45 on dist-test-slave-w4v5"
server_key: "24d3ebb873b11b1efab2f4851a4f3a57"
server_key_iv: "1358d968348eec80b79ad43cb5458b64"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:45.970633 21388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:13:45.986294 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:45.987965 21388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:13:45.988130 21388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "a4705b50b1e44cbcb975fa05fb10b35e"
format_stamp: "Formatted at 2026-08-06 09:13:45 on dist-test-slave-w4v5"
server_key: "24d3ebb873b11b1efab2f4851a4f3a57"
server_key_iv: "1358d968348eec80b79ad43cb5458b64"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:45.988332 21388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:46.011135 21388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:46.014956 21388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:46.015295 21388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:46.016348 21388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:13:46.017661 21388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:13:46.017738 21388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:46.017822 21388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:13:46.017867 21388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:46.053457 21388 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:39675
I20260806 09:13:46.053494 21574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:39675 every 8 connection(s)
I20260806 09:13:46.054838 21388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:13:46.057055 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 21388
I20260806 09:13:46.057246 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:13:46.057822 3348 external_mini_cluster.cc:1428] Setting key 0ef9c192599b3134d098deaf3065107d
Aug 06 09:13:46 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007625, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:46.075513 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:46.057508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:52331 (local address 127.3.69.62:36197)
0806 09:13:46.058161 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:46.058167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:46.059218 (+ 1051us) server_negotiation.cc:407] Connection header received
0806 09:13:46.061554 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:46.061559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:46.061671 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:46.061791 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:46.063198 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.063920 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.065493 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.065918 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.066056 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:46.069410 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:46.069415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:46.069418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:46.069450 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:46.072339 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:46.073173 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:46.073180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:46.073183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:46.073372 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:46.073912 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:46.073916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:46.073918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:46.074194 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:46.074388 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:46.074994 (+ 606us) server_negotiation.cc:299] Negotiation successful
0806 09:13:46.075243 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":281,"threads_started":1}
I20260806 09:13:46.076556 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:46.058138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36197 (local address 127.3.69.3:52331)
0806 09:13:46.059000 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:46.059045 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:46.061291 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:46.062024 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:46.062032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:46.062444 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:46.062991 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:46.063007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.064092 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.064103 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:46.065342 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:46.065354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.066162 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.066169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:46.066396 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:46.067043 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:46.067147 (+ 104us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:46.069182 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:46.072471 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:46.072479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:46.072495 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:46.073008 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:46.073487 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:46.073491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:46.073495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:46.073792 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:46.074364 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:46.074370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:46.074670 (+ 300us) client_negotiation.cc:769] Sending connection context
0806 09:13:46.074891 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 09:13:46.075333 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":346,"threads_started":1}
I20260806 09:13:46.077957 21576 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36197
I20260806 09:13:46.078290 21576 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:46.078989 21576 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
I20260806 09:13:46.080456 20837 ts_manager.cc:194] Registered new tserver with Master: a4705b50b1e44cbcb975fa05fb10b35e (127.3.69.3:39675)
I20260806 09:13:46.081254 20837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:52331
I20260806 09:13:46.086325 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:13:46.088152 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 20957
Aug 06 09:13:46 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:13:46.105579 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:46.089104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.3.69.1:42027)
0806 09:13:46.089830 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:46.089836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:46.089859 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:13:46.089919 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:46.089925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:46.090181 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:46.090345 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:46.091954 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.093155 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.094659 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.095272 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.095368 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:46.099829 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:46.099842 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:46.099854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:46.099890 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:46.102847 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:46.103573 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:46.103580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:46.103583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:46.103740 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:46.104199 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:46.104202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:46.104204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:46.104554 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:46.104797 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:46.105083 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 09:13:46.105298 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":447,"threads_started":1}
I20260806 09:13:46.107012 21156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:46.193598 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 20957
I20260806 09:13:46.218870 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 21179
Aug 06 09:13:46 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:13:46.241998 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:46.219951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.3.69.2:33813)
0806 09:13:46.220525 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:46.220532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:46.220558 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:13:46.220619 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:46.220626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:46.220847 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:46.220977 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:46.222226 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.223393 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.224730 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.225285 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.225392 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:46.235258 (+ 9866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:46.235275 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:46.235293 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:46.235346 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:46.238773 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:46.239655 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:46.239664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:46.239666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:46.239842 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:46.240206 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:46.240210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:46.240212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:46.240656 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:46.240925 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:46.241256 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 09:13:46.241501 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":294,"threads_started":1}
I20260806 09:13:46.243403 21370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:46.326697 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 21179
I20260806 09:13:46.349470 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 21388
Aug 06 09:13:46 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:13:46.367235 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:46.350413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.3.69.3:39675)
0806 09:13:46.351170 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:46.351178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:46.351209 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:13:46.351274 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:46.351281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:46.351514 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:46.351664 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:46.352990 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.354055 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.355549 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:46.356142 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:46.356260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:46.361153 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:46.361166 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:46.361181 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:46.361218 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:46.364211 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:46.364966 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:46.364974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:46.364977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:46.365105 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:46.365501 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:46.365506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:46.365509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:46.365869 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:46.366212 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:46.366518 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 09:13:46.366808 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":349,"threads_started":1}
I20260806 09:13:46.368512 21567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:46.454181 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 21388
I20260806 09:13:46.478021 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 20786
I20260806 09:13:46.479637 20886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:46.587117 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 20786
I20260806 09:13:46.620504 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36197
--webserver_interface=127.3.69.62
--webserver_port=43633
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:36197
--ranger_config_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:58635/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.69.20:58635/kms with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:13:46.966629 21597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:46.966933 21597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:46.967160 21597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:46.975149 21597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:13:46.975289 21597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:46.975371 21597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:46.975449 21597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:13:46.975540 21597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:13:46.986369 21597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.69.20:58635/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:36197
--ranger_config_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:46.987808 21597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:46.989161 21597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:47.001232 21603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:47.001157 21604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:47.001396 21606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:47.002169 21597 server_base.cc:1034] running on GCE node
I20260806 09:13:47.002948 21597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:47.004514 21597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:47.005733 21597 hybrid_clock.cc:648] HybridClock initialized: now 1786007627005689 us; error 39 us; skew 500 ppm
I20260806 09:13:47.006135 21597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:47 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:47.009284 21597 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:13:47.010805 21597 webserver.cc:460] Webserver started at http://127.3.69.62:43633/ using document root <none> and password file <none>
I20260806 09:13:47.011531 21597 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:47.011647 21597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:13:47 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:47.096027 21597 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:13:47.099207 21623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:47.100617 21597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:13:47.100826 21597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "e794ec9bf3904b8eb769204b60660c16"
format_stamp: "Formatted at 2026-08-06 09:13:42 on dist-test-slave-w4v5"
server_key: "0095fc3d6e4dc46a3e8b31aa9130e7a0"
server_key_iv: "6bd2b462cef6b5874ab1284cc979cbe9"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:47.101801 21597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:47.122773 21597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:47.126845 21597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:47.127215 21597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:47.147256 21597 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:36197
I20260806 09:13:47.147276 21699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:36197 every 8 connection(s)
I20260806 09:13:47.148752 21597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:13:47.155664 21700 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:13:47.159020 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Bootstrap starting.
I20260806 09:13:47.159193 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 21597
I20260806 09:13:47.160265 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:58635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:42027
--local_ip_for_outbound_sockets=127.3.69.1
--tserver_master_addrs=127.3.69.62:36197
--webserver_port=36657
--webserver_interface=127.3.69.1
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.69.20:58635/kms with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:13:47.181777 21700 log.cc:824] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Log is configured to *not* fsync() on all Append() calls
I20260806 09:13:47.191354 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:13:47.191922 21700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Bootstrap complete.
I20260806 09:13:47.201568 21700 raft_consensus.cc:348] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } }
I20260806 09:13:47.202438 21700 raft_consensus.cc:729] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e794ec9bf3904b8eb769204b60660c16, State: Initialized, Role: FOLLOWER
I20260806 09:13:47.203191 21700 consensus_queue.cc:260] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } }
I20260806 09:13:47.203434 21700 raft_consensus.cc:388] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:13:47.203581 21700 raft_consensus.cc:482] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:13:47.203789 21700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:13:47.208072 21700 raft_consensus.cc:504] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } }
I20260806 09:13:47.208576 21700 leader_election.cc:302] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e794ec9bf3904b8eb769204b60660c16; no voters:
I20260806 09:13:47.209218 21700 leader_election.cc:288] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:13:47.209434 21704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:13:47.210446 21704 raft_consensus.cc:686] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [term 2 LEADER]: Becoming Leader. State: Replica: e794ec9bf3904b8eb769204b60660c16, State: Running, Role: LEADER
I20260806 09:13:47.211126 21704 consensus_queue.cc:237] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } }
I20260806 09:13:47.211400 21700 sys_catalog.cc:564] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:13:47.214036 21706 sys_catalog.cc:455] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e794ec9bf3904b8eb769204b60660c16. Latest consensus state: current_term: 2 leader_uuid: "e794ec9bf3904b8eb769204b60660c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } } }
I20260806 09:13:47.214071 21705 sys_catalog.cc:455] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e794ec9bf3904b8eb769204b60660c16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e794ec9bf3904b8eb769204b60660c16" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36197 } } }
I20260806 09:13:47.214550 21705 sys_catalog.cc:458] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: This master's current role is: LEADER
I20260806 09:13:47.214558 21706 sys_catalog.cc:458] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16 [sys.catalog]: This master's current role is: LEADER
I20260806 09:13:47.228848 21711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:13:47.238015 21711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:13:47.238905 21700 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 09:13:47.239622 21711 catalog_manager.cc:1259] Loaded cluster ID: fbba69464fd34e6cad7136c50ecde290
I20260806 09:13:47.239813 21711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:13:47.243672 21711 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:13:47.245191 21711 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Loaded TSK: 0
I20260806 09:13:47.246868 21711 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:13:47.607024 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:47.607554 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:47.607779 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:47.619837 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:47.620069 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:47.620347 21702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:13:47.637454 21702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.69.20:58635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:42027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=36657
--tserver_master_addrs=127.3.69.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:47.639533 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:47.641386 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:47.673094 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:47.675256 21756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:47.678287 21760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:47.679474 21702 server_base.cc:1034] running on GCE node
I20260806 09:13:47.680378 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:47.681769 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:47.683100 21702 hybrid_clock.cc:648] HybridClock initialized: now 1786007627682954 us; error 106 us; skew 500 ppm
I20260806 09:13:47.683506 21702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:47 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:47.691334 21702 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:13:47.704370 21702 webserver.cc:460] Webserver started at http://127.3.69.1:36657/ using document root <none> and password file <none>
I20260806 09:13:47.705381 21702 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:47.705590 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:13:47 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:47.739024 21702 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 09:13:47.742516 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:47.744343 21702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:13:47.744524 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "8c6df30184d2490cbf4a5889b932b0f9"
format_stamp: "Formatted at 2026-08-06 09:13:44 on dist-test-slave-w4v5"
server_key: "3cde8514e1b6d3f95441e32f08589858"
server_key_iv: "623534543b504383b53b072df2b25a8f"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:47.745972 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:47.790154 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:47.795849 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:47.796231 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:47.798542 21702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:13:47.806628 21702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:13:47.806759 21702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:47.806875 21702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:13:47.806951 21702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:47.862165 21702 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:42027
I20260806 09:13:47.862259 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:42027 every 8 connection(s)
I20260806 09:13:47.864056 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:13:47.868317 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 21702
I20260806 09:13:47.869107 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:58635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:33813
--local_ip_for_outbound_sockets=127.3.69.2
--tserver_master_addrs=127.3.69.62:36197
--webserver_port=34613
--webserver_interface=127.3.69.2
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.69.20:58635/kms with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:13:47 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:47.900616 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:47.867572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36197 (local address 127.3.69.1:38585)
0806 09:13:47.871936 (+ 4364us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:47.871984 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:47.878974 (+ 6990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:47.881755 (+ 2781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:47.881774 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:47.882836 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:47.883653 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:47.883679 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:47.886099 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:47.886104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:47.888139 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:47.888164 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:47.889561 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:47.889570 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:47.889833 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:47.890602 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:47.890640 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:47.892943 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:47.896339 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:47.896347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:47.896361 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:47.896767 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:47.897217 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:47.897224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:47.897227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:47.897433 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:47.898315 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:47.898324 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:47.898648 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 09:13:47.898923 (+ 275us) client_negotiation.cc:240] Negotiation successful
0806 09:13:47.899497 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4069,"thread_start_us":3834,"threads_started":1}
I20260806 09:13:47.902693 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:47.868424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:38585 (local address 127.3.69.62:36197)
0806 09:13:47.879269 (+ 10845us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:47.879289 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:47.879340 (+ 51us) server_negotiation.cc:407] Connection header received
0806 09:13:47.879706 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:47.879718 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:47.880070 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:47.880852 (+ 782us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:47.883928 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:47.885884 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:47.888331 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:47.889311 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:47.889508 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:47.893215 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:47.893222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:47.893236 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:47.893279 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:47.896180 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:47.896929 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:47.896936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:47.896942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:47.897097 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:47.897571 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:47.897574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:47.897577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:47.898165 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:47.898387 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:47.901385 (+ 2998us) server_negotiation.cc:299] Negotiation successful
0806 09:13:47.901683 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10518,"thread_start_us":11686,"threads_started":1}
I20260806 09:13:47.905705 21951 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36197
I20260806 09:13:47.906069 21951 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:47.906870 21951 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
W20260806 09:13:47.909901 21951 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:36197 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:13:47.910143 21951 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:47.910414 21951 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
I20260806 09:13:47.911581 21951 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:47.911845 21951 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
W20260806 09:13:47.912935 21951 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:13:48.296640 21958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:48.297032 21958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:48.297248 21958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:48.308986 21958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:48.309139 21958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:48.309330 21958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:13:48.321105 21958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.69.20:58635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:33813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=34613
--tserver_master_addrs=127.3.69.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:48.322537 21958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:48.323949 21958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:48.344251 21977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:48.344748 21978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:48.352216 21985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:48.352774 21958 server_base.cc:1034] running on GCE node
I20260806 09:13:48.353682 21958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:48.355149 21958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:48.356472 21958 hybrid_clock.cc:648] HybridClock initialized: now 1786007628356402 us; error 64 us; skew 500 ppm
I20260806 09:13:48.356841 21958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:48 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:48.360226 21958 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:13:48.362141 21958 webserver.cc:460] Webserver started at http://127.3.69.2:34613/ using document root <none> and password file <none>
I20260806 09:13:48.362892 21958 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:48.363065 21958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:13:48 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:48.390128 21958 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:13:48.395776 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:48.397671 21958 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260806 09:13:48.397969 21958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "ce6c18bfeabd4fbdbb73a956c1705d19"
format_stamp: "Formatted at 2026-08-06 09:13:45 on dist-test-slave-w4v5"
server_key: "be268c1dc69d36d1e9dbb33b9d14680c"
server_key_iv: "70723c74f7cd6e4cd89f501d80dc2960"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:48.399530 21958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:48.461486 21958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:48.466957 21958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:48.467347 21958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:48.473172 21958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:13:48.475165 21958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:13:48.475322 21958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:48.475481 21958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:13:48.475611 21958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:48.535701 21958 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:33813
I20260806 09:13:48.535763 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:33813 every 8 connection(s)
I20260806 09:13:48.537766 21958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:13:48.539498 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 21958
I20260806 09:13:48.540277 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.69.20:58635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:39675
--local_ip_for_outbound_sockets=127.3.69.3
--tserver_master_addrs=127.3.69.62:36197
--webserver_port=45261
--webserver_interface=127.3.69.3
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.69.20:58635/kms with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:13:48 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:48.576881 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:48.550149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:52525 (local address 127.3.69.62:36197)
0806 09:13:48.554084 (+ 3935us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:48.554091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:48.557475 (+ 3384us) server_negotiation.cc:407] Connection header received
0806 09:13:48.557565 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:48.557571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:48.557667 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:48.557780 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:48.559978 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:48.561192 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:48.563688 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:48.564384 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:48.564479 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:48.569075 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:48.569082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:48.569087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:48.569130 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:48.573017 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:48.573980 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:48.573985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:48.573998 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:48.574175 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:48.574751 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:48.574756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:48.574760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:48.575159 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:48.575296 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:48.576252 (+ 956us) server_negotiation.cc:299] Negotiation successful
0806 09:13:48.576481 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3798,"thread_start_us":3613,"threads_started":1}
I20260806 09:13:48.580209 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:48.549463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36197 (local address 127.3.69.2:52525)
0806 09:13:48.554660 (+ 5197us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:48.554715 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:48.557114 (+ 2399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:48.558246 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:48.558259 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:48.558914 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:48.559767 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:48.559791 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:48.561407 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:48.561412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:48.563494 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:48.563509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:48.564858 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:48.564865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:48.565139 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:48.565930 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:48.565962 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:48.568807 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:48.573186 (+ 4379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:48.573196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:48.573213 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:48.573817 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:48.574331 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:48.574336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:48.574340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:48.574611 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:48.575389 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:48.575397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:48.575812 (+ 415us) client_negotiation.cc:769] Sending connection context
0806 09:13:48.577763 (+ 1951us) client_negotiation.cc:240] Negotiation successful
0806 09:13:48.578376 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4887,"thread_start_us":4633,"threads_started":1}
I20260806 09:13:48.582366 22169 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36197
I20260806 09:13:48.582692 22169 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:48.583804 22169 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
I20260806 09:13:48.585958 21649 ts_manager.cc:194] Registered new tserver with Master: ce6c18bfeabd4fbdbb73a956c1705d19 (127.3.69.2:33813)
I20260806 09:13:48.588140 21649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:52525
Aug 06 09:13:48 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786007628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:48.913093 21951 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:48.913591 21951 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
I20260806 09:13:48.915002 21649 ts_manager.cc:194] Registered new tserver with Master: 8c6df30184d2490cbf4a5889b932b0f9 (127.3.69.1:42027)
I20260806 09:13:48.916333 21649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:38585
W20260806 09:13:48.990633 22174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:48.991108 22174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:48.991291 22174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:49.000970 22174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:49.001118 22174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:49.001309 22174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:13:49.014057 22174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.69.20:58635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:39675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=45261
--tserver_master_addrs=127.3.69.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:49.015574 22174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:49.016945 22174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:49.035532 22182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:49.035532 22183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:49.040537 22187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:49.046268 22174 server_base.cc:1034] running on GCE node
I20260806 09:13:49.047022 22174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:49.048580 22174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:49.049865 22174 hybrid_clock.cc:648] HybridClock initialized: now 1786007629049784 us; error 63 us; skew 500 ppm
I20260806 09:13:49.050321 22174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:49 dist-test-slave-w4v5 krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:49.061110 22174 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
Aug 06 09:13:49 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:49.076628 22174 webserver.cc:460] Webserver started at http://127.3.69.3:45261/ using document root <none> and password file <none>
I20260806 09:13:49.077629 22174 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:49.078091 22174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:13:49 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:49.142135 22174 fs_manager.cc:709] Time spent opening directory manager: real 0.028s user 0.006s sys 0.005s
I20260806 09:13:49.157007 22197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:49.159093 22174 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.010s sys 0.004s
I20260806 09:13:49.159308 22174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "a4705b50b1e44cbcb975fa05fb10b35e"
format_stamp: "Formatted at 2026-08-06 09:13:45 on dist-test-slave-w4v5"
server_key: "24d3ebb873b11b1efab2f4851a4f3a57"
server_key_iv: "1358d968348eec80b79ad43cb5458b64"
server_key_version: "kuduclusterkey@0"
I20260806 09:13:49.160948 22174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:49.232388 22174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:49.238018 22174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:49.238348 22174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:49.239754 22174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:13:49.241734 22174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:13:49.241863 22174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:49.241992 22174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:13:49.242084 22174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:49.296109 22174 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:39675
I20260806 09:13:49.296176 22350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:39675 every 8 connection(s)
I20260806 09:13:49.298099 22174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:13:49.303054 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 22174
Aug 06 09:13:49 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:49.348445 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.304064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.3.69.62:36197)
0806 09:13:49.328969 (+ 24905us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.328975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.328995 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:13:49.329046 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.329050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.329162 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.329274 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:49.331840 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.333071 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.335128 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.335724 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.335844 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.339565 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:49.339571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:49.339575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:49.339611 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:49.343693 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:49.344920 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:49.344930 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:49.344934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:49.345102 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:49.345890 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:49.345896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:49.345899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:49.346283 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:49.347656 (+ 1373us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.347786 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.348022 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":24784,"spinlock_wait_cycles":5632,"thread_start_us":24608,"threads_started":1}
I20260806 09:13:49.352119 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.301683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:51063 (local address 127.3.69.62:36197)
0806 09:13:49.328669 (+ 26986us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.328679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.328702 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:13:49.328789 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.328795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.328904 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.329020 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:49.331794 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.333068 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.338279 (+ 5211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.338992 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.339169 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.343609 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:49.343617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:49.343623 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:49.343673 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:49.347206 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:49.348412 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:49.348417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:49.348420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:49.348599 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:49.349202 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:49.349208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:49.349210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:49.349574 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:49.349719 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.351243 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.351469 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26797,"spinlock_wait_cycles":14848,"thread_start_us":26635,"threads_started":1}
I20260806 09:13:49.354674 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.301937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36197 (local address 127.3.69.3:51063)
0806 09:13:49.323937 (+ 22000us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:49.324001 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:49.327728 (+ 3727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:49.330086 (+ 2358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:49.330099 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:49.330722 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:49.331574 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.331597 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.335236 (+ 3639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.335242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.338084 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.338103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.339478 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.339484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.339777 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.340528 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:49.340569 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:49.343280 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:49.347581 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:49.347591 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:49.347612 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:49.348266 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:49.348740 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:49.348745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:49.348748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:49.349054 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:49.349795 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:49.349804 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:49.350256 (+ 452us) client_negotiation.cc:769] Sending connection context
0806 09:13:49.350597 (+ 341us) client_negotiation.cc:240] Negotiation successful
0806 09:13:49.352684 (+ 2087us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21613,"thread_start_us":22313,"threads_started":1}
I20260806 09:13:49.356357 22353 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36197
I20260806 09:13:49.356745 22353 heartbeater.cc:464] Registering TS with master...
I20260806 09:13:49.357661 22353 heartbeater.cc:511] Master 127.3.69.62:36197 requested a full tablet report, sending...
I20260806 09:13:49.359432 21649 ts_manager.cc:194] Registered new tserver with Master: a4705b50b1e44cbcb975fa05fb10b35e (127.3.69.3:39675)
I20260806 09:13:49.360464 21649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:51063
I20260806 09:13:49.366370 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:13:49.397284 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.373746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.3.69.62:36197)
0806 09:13:49.373940 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.373949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.379709 (+ 5760us) server_negotiation.cc:407] Connection header received
0806 09:13:49.379878 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.379884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.379993 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.380115 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:49.382142 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.383259 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.385446 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.386121 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.386254 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.390091 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:49.390099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:49.390103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:49.390158 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:49.393872 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:49.394756 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:49.394761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:49.394764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:49.394919 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:49.395427 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:49.395432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:49.395435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:49.395821 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:49.396047 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.396687 (+ 640us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.396930 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:13:49.404390 21649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:13:49.416337 21649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 06 09:13:49 dist-test-slave-w4v5 krb5kdc[20168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for HTTP/127.3.69.20@KRBTEST.COM
I20260806 09:13:49.487164 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.447577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42027 (local address 127.0.0.1:43764)
0806 09:13:49.477721 (+ 30144us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:49.477745 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:49.477876 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:49.478819 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:49.478826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:49.478881 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:13:49.479406 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.479421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.481549 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.481555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.483901 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.483917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.486144 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.486151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.486356 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.486406 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 09:13:49.486575 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 09:13:49.486770 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30018,"thread_start_us":29872,"threads_started":1}
I20260806 09:13:49.487504 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.447999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.3.69.1:42027)
0806 09:13:49.464234 (+ 16235us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.464248 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.477852 (+ 13604us) server_negotiation.cc:407] Connection header received
0806 09:13:49.478032 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.478043 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.478304 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.478503 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:13:49.479649 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.481372 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.484098 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.485561 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.485666 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.485896 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.486679 (+ 783us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.486859 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15946,"thread_start_us":16253,"threads_started":1}
I20260806 09:13:49.493625 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.457068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:39675 (local address 127.0.0.1:54424)
0806 09:13:49.478374 (+ 21306us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:49.478393 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:49.478578 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:49.479619 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:49.479623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:49.479659 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:13:49.479985 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.479996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.483320 (+ 3324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.483329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.487053 (+ 3724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.487089 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.492000 (+ 4911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.492007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.493011 (+ 1004us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.493040 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:13:49.493119 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 09:13:49.493248 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21184,"mutex_wait_us":819,"thread_start_us":18682,"threads_started":1}
I20260806 09:13:49.494469 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.457805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.3.69.3:39675)
0806 09:13:49.477261 (+ 19456us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.477271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.478474 (+ 1203us) server_negotiation.cc:407] Connection header received
0806 09:13:49.478800 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.478808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.479041 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.479240 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:13:49.480229 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.483130 (+ 2901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.489884 (+ 6754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.491459 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.491572 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.491788 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.493927 (+ 2139us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.494122 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19267,"thread_start_us":19122,"threads_started":1}
I20260806 09:13:49.497138 21836 tablet_service.cc:1467] Processing CreateTablet for tablet 818c9b17bdcb404683f3d2d759b0543d (DEFAULT_TABLE table=test-table [id=129a34b94df54849b92cbc0d03fb121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:13:49.498270 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.445958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.3.69.2:33813)
0806 09:13:49.475566 (+ 29608us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.475578 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.476436 (+ 858us) server_negotiation.cc:407] Connection header received
0806 09:13:49.476808 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.476820 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.477124 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.477337 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:13:49.489160 (+ 11823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.490678 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.492986 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.494440 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.494588 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.494784 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.497740 (+ 2956us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.497923 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29434,"thread_start_us":29330,"threads_started":1}
I20260806 09:13:49.498136 22270 tablet_service.cc:1467] Processing CreateTablet for tablet 818c9b17bdcb404683f3d2d759b0543d (DEFAULT_TABLE table=test-table [id=129a34b94df54849b92cbc0d03fb121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:13:49.498786 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818c9b17bdcb404683f3d2d759b0543d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:13:49.499116 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.445805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:33813 (local address 127.0.0.1:40716)
0806 09:13:49.476244 (+ 30439us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:49.476295 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:49.476547 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:49.487640 (+ 11093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:49.487645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:49.487671 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:13:49.488952 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.488964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.490864 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.490868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.492794 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.492810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.497310 (+ 4500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.497317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.497501 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.497533 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:13:49.498525 (+ 992us) client_negotiation.cc:240] Negotiation successful
0806 09:13:49.498692 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30094,"mutex_wait_us":1,"thread_start_us":32297,"threads_started":1}
I20260806 09:13:49.499559 22270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818c9b17bdcb404683f3d2d759b0543d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:13:49.516248 22070 tablet_service.cc:1467] Processing CreateTablet for tablet 818c9b17bdcb404683f3d2d759b0543d (DEFAULT_TABLE table=test-table [id=129a34b94df54849b92cbc0d03fb121a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:13:49.517753 22070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818c9b17bdcb404683f3d2d759b0543d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:13:49.581858 22383 tablet_bootstrap.cc:492] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: Bootstrap starting.
I20260806 09:13:49.583938 22382 tablet_bootstrap.cc:492] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Bootstrap starting.
I20260806 09:13:49.588204 22383 tablet_bootstrap.cc:654] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: Neither blocks nor log segments found. Creating new log.
I20260806 09:13:49.588274 22382 tablet_bootstrap.cc:654] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Neither blocks nor log segments found. Creating new log.
I20260806 09:13:49.589602 22382 log.cc:824] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:13:49.593806 22169 heartbeater.cc:503] Master 127.3.69.62:36197 was elected leader, sending a full tablet report...
I20260806 09:13:49.606101 22383 log.cc:824] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:13:49.609324 22382 tablet_bootstrap.cc:492] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: No bootstrap required, opened a new log
I20260806 09:13:49.609628 22382 ts_tablet_manager.cc:1397] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.019s
I20260806 09:13:49.618098 22383 tablet_bootstrap.cc:492] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: No bootstrap required, opened a new log
I20260806 09:13:49.618449 22383 ts_tablet_manager.cc:1397] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.009s
I20260806 09:13:49.619223 22385 tablet_bootstrap.cc:492] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Bootstrap starting.
I20260806 09:13:49.620636 22382 raft_consensus.cc:348] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.621152 22382 raft_consensus.cc:374] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:13:49.621272 22382 raft_consensus.cc:729] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4705b50b1e44cbcb975fa05fb10b35e, State: Initialized, Role: FOLLOWER
I20260806 09:13:49.622016 22382 consensus_queue.cc:260] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.623340 22385 tablet_bootstrap.cc:654] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Neither blocks nor log segments found. Creating new log.
I20260806 09:13:49.624953 22385 log.cc:824] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Log is configured to *not* fsync() on all Append() calls
I20260806 09:13:49.631350 22382 ts_tablet_manager.cc:1428] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Time spent starting tablet: real 0.022s user 0.013s sys 0.007s
I20260806 09:13:49.631758 22353 heartbeater.cc:503] Master 127.3.69.62:36197 was elected leader, sending a full tablet report...
I20260806 09:13:49.633859 22383 raft_consensus.cc:348] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.634419 22383 raft_consensus.cc:374] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:13:49.634593 22383 raft_consensus.cc:729] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6df30184d2490cbf4a5889b932b0f9, State: Initialized, Role: FOLLOWER
I20260806 09:13:49.635428 22383 consensus_queue.cc:260] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.638725 22385 tablet_bootstrap.cc:492] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: No bootstrap required, opened a new log
I20260806 09:13:49.639040 22385 ts_tablet_manager.cc:1397] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.010s
I20260806 09:13:49.639039 21951 heartbeater.cc:503] Master 127.3.69.62:36197 was elected leader, sending a full tablet report...
I20260806 09:13:49.640033 22383 ts_tablet_manager.cc:1428] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: Time spent starting tablet: real 0.021s user 0.020s sys 0.000s
I20260806 09:13:49.648247 22385 raft_consensus.cc:348] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.648739 22385 raft_consensus.cc:374] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:13:49.648882 22385 raft_consensus.cc:729] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6c18bfeabd4fbdbb73a956c1705d19, State: Initialized, Role: FOLLOWER
I20260806 09:13:49.649564 22385 consensus_queue.cc:260] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.651496 22385 ts_tablet_manager.cc:1428] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 09:13:49.693356 22392 raft_consensus.cc:482] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:13:49.693747 22392 raft_consensus.cc:504] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.695982 22392 leader_election.cc:288] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c6df30184d2490cbf4a5889b932b0f9 (127.3.69.1:42027), a4705b50b1e44cbcb975fa05fb10b35e (127.3.69.3:39675)
I20260806 09:13:49.697885 22387 raft_consensus.cc:482] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:13:49.698297 22387 raft_consensus.cc:504] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.700697 22387 leader_election.cc:288] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce6c18bfeabd4fbdbb73a956c1705d19 (127.3.69.2:33813), 8c6df30184d2490cbf4a5889b932b0f9 (127.3.69.1:42027)
I20260806 09:13:49.711340 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.701257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:33813 (local address 127.3.69.3:53179)
0806 09:13:49.701475 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:49.701512 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:49.701667 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:49.703962 (+ 2295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:49.703968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:49.704000 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:13:49.704230 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.704243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.706099 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.706106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.708342 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.708359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.710430 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.710436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.710691 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.710730 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:13:49.710832 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 09:13:49.710975 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 09:13:49.712687 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.703185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:53179 (local address 127.3.69.2:33813)
0806 09:13:49.703438 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.703445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.703468 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:13:49.703561 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.703567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.703653 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.703762 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:13:49.704430 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.705809 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.708577 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.710082 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.710230 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.710335 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.712209 (+ 1874us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.712356 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:13:49.713694 22105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818c9b17bdcb404683f3d2d759b0543d" candidate_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" is_pre_election: true
I20260806 09:13:49.714275 22105 raft_consensus.cc:2454] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4705b50b1e44cbcb975fa05fb10b35e in term 0.
I20260806 09:13:49.715116 22204 leader_election.cc:302] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4705b50b1e44cbcb975fa05fb10b35e, ce6c18bfeabd4fbdbb73a956c1705d19; no voters:
I20260806 09:13:49.715622 22387 raft_consensus.cc:2781] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:13:49.715790 22387 raft_consensus.cc:482] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:13:49.715919 22387 raft_consensus.cc:3037] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:13:49.719594 22387 raft_consensus.cc:504] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.720791 22105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818c9b17bdcb404683f3d2d759b0543d" candidate_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19"
I20260806 09:13:49.721042 22105 raft_consensus.cc:3037] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:13:49.723686 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.696917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:52591 (local address 127.3.69.1:42027)
0806 09:13:49.697142 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.697148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.709575 (+ 12427us) server_negotiation.cc:407] Connection header received
0806 09:13:49.710662 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.710671 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.710805 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.710880 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:13:49.712473 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.713798 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.719845 (+ 6047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.721530 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.721677 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.721861 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.723251 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.723445 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:13:49.725069 22105 raft_consensus.cc:2454] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4705b50b1e44cbcb975fa05fb10b35e in term 1.
I20260806 09:13:49.725544 22204 leader_election.cc:302] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4705b50b1e44cbcb975fa05fb10b35e, ce6c18bfeabd4fbdbb73a956c1705d19; no voters:
I20260806 09:13:49.729676 22387 leader_election.cc:288] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [CANDIDATE]: Term 1 election: Requested vote from peers ce6c18bfeabd4fbdbb73a956c1705d19 (127.3.69.2:33813), 8c6df30184d2490cbf4a5889b932b0f9 (127.3.69.1:42027)
I20260806 09:13:49.729907 22387 raft_consensus.cc:2781] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:13:49.730077 22387 raft_consensus.cc:686] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 1 LEADER]: Becoming Leader. State: Replica: a4705b50b1e44cbcb975fa05fb10b35e, State: Running, Role: LEADER
I20260806 09:13:49.730695 22387 consensus_queue.cc:237] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } }
I20260806 09:13:49.732000 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.701919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42027 (local address 127.3.69.3:55051)
0806 09:13:49.711935 (+ 10016us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:49.711954 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:49.712053 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:49.724451 (+ 12398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:49.724457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:49.724487 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:13:49.724680 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.724692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.726187 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.726193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.728381 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.728403 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.730899 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.730909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.731222 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.731275 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 09:13:49.731390 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 09:13:49.731535 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9912,"threads_started":1}
I20260806 09:13:49.733263 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.702768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:55051 (local address 127.3.69.1:42027)
0806 09:13:49.723920 (+ 21152us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.723925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.723943 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:13:49.723998 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.724003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.724072 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.724179 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:13:49.724868 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.726020 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.728676 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.730432 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.730602 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.730771 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.732202 (+ 1431us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.732365 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21053,"threads_started":1}
I20260806 09:13:49.734728 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.696327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:42027 (local address 127.3.69.2:52591)
0806 09:13:49.709436 (+ 13109us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:49.709463 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:49.710304 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:49.711160 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:49.711167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:49.711201 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:13:49.711609 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.711622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.714032 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.714041 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.716723 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.716743 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.722126 (+ 5383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.722134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.722387 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.722438 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 09:13:49.733453 (+ 11015us) client_negotiation.cc:240] Negotiation successful
0806 09:13:49.733638 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12966,"thread_start_us":12758,"threads_started":1}
I20260806 09:13:49.738073 21879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818c9b17bdcb404683f3d2d759b0543d" candidate_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c6df30184d2490cbf4a5889b932b0f9" is_pre_election: true
I20260806 09:13:49.738684 21879 raft_consensus.cc:2454] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4705b50b1e44cbcb975fa05fb10b35e in term 0.
I20260806 09:13:49.739423 21879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818c9b17bdcb404683f3d2d759b0543d" candidate_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c6df30184d2490cbf4a5889b932b0f9"
I20260806 09:13:49.739619 21879 raft_consensus.cc:3037] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:13:49.742883 21879 raft_consensus.cc:2454] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4705b50b1e44cbcb975fa05fb10b35e in term 1.
I20260806 09:13:49.743382 21879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818c9b17bdcb404683f3d2d759b0543d" candidate_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c6df30184d2490cbf4a5889b932b0f9" is_pre_election: true
I20260806 09:13:49.743620 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.697199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:55855 (local address 127.3.69.3:39675)
0806 09:13:49.697397 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.697405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.735366 (+ 37961us) server_negotiation.cc:407] Connection header received
0806 09:13:49.736032 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.736038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.736111 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.736177 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:13:49.737368 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.738383 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.741053 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.742301 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.742396 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.742491 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.743163 (+ 672us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.743316 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 09:13:49.743700 21879 raft_consensus.cc:2379] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ce6c18bfeabd4fbdbb73a956c1705d19 in current term 1: Already voted for candidate a4705b50b1e44cbcb975fa05fb10b35e in this term.
I20260806 09:13:49.744372 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.696667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:39675 (local address 127.3.69.2:55855)
0806 09:13:49.735269 (+ 38602us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:49.735290 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:49.735696 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:49.736389 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:49.736394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:49.736421 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:13:49.736605 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.736616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.738601 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.738606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.740870 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:49.740886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.742696 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.742704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:49.742928 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.742970 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 09:13:49.743907 (+ 937us) client_negotiation.cc:240] Negotiation successful
0806 09:13:49.744058 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38482,"thread_start_us":40268,"threads_started":1}
I20260806 09:13:49.745391 22299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818c9b17bdcb404683f3d2d759b0543d" candidate_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" is_pre_election: true
I20260806 09:13:49.746431 21998 leader_election.cc:302] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce6c18bfeabd4fbdbb73a956c1705d19; no voters: 8c6df30184d2490cbf4a5889b932b0f9, a4705b50b1e44cbcb975fa05fb10b35e
I20260806 09:13:49.747155 22392 raft_consensus.cc:2726] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 09:13:49.749905 21646 catalog_manager.cc:5495] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e reported cstate change: term changed from 0 to 1, leader changed from <none> to a4705b50b1e44cbcb975fa05fb10b35e (127.3.69.3). New cstate: current_term: 1 leader_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4705b50b1e44cbcb975fa05fb10b35e" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 39675 } health_report { overall_health: HEALTHY } } }
W20260806 09:13:49.796466 22170 tablet.cc:2369] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:13:49.801573 22354 tablet.cc:2369] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:13:49.827270 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.820412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.3.69.3:39675)
0806 09:13:49.820630 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.820637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.820741 (+ 104us) server_negotiation.cc:407] Connection header received
0806 09:13:49.820951 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.820958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.821094 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.821282 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:13:49.821835 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.822631 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.824716 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.825128 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.825239 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.825910 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:13:49.826456 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:13:49.826648 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.826780 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.826872 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:13:49.835785 21879 raft_consensus.cc:1261] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 1 FOLLOWER]: Refusing update from remote peer a4705b50b1e44cbcb975fa05fb10b35e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:13:49.836009 22105 raft_consensus.cc:1261] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 1 FOLLOWER]: Refusing update from remote peer a4705b50b1e44cbcb975fa05fb10b35e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:13:49.836809 22399 consensus_queue.cc:1035] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c6df30184d2490cbf4a5889b932b0f9" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 42027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:13:49.837555 22387 consensus_queue.cc:1035] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6c18bfeabd4fbdbb73a956c1705d19" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 33813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:13:49.850935 22410 mvcc.cc:205] Tried to move back new op lower bound from 7315487251797094400 to 7315487251388690432. Current Snapshot: MvccSnapshot[applied={T|T < 7315487251797094400}]
I20260806 09:13:49.881425 21646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:13:49.881764 21646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:13:49.886205 21646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e794ec9bf3904b8eb769204b60660c16: Sending DeleteTablet for 3 replicas of tablet 818c9b17bdcb404683f3d2d759b0543d
I20260806 09:13:49.887768 21836 tablet_service.cc:1514] Processing DeleteTablet for tablet 818c9b17bdcb404683f3d2d759b0543d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:13:49 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:43764
I20260806 09:13:49.888477 22270 tablet_service.cc:1514] Processing DeleteTablet for tablet 818c9b17bdcb404683f3d2d759b0543d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:13:49 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:54424
I20260806 09:13:49.889128 22070 tablet_service.cc:1514] Processing DeleteTablet for tablet 818c9b17bdcb404683f3d2d759b0543d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:13:49 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:40716
I20260806 09:13:49.890594 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 21702
I20260806 09:13:49.899771 22424 tablet_replica.cc:330] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: stopping tablet replica
I20260806 09:13:49.900465 22423 tablet_replica.cc:330] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: stopping tablet replica
I20260806 09:13:49.901101 22422 tablet_replica.cc:330] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: stopping tablet replica
I20260806 09:13:49.901602 22422 raft_consensus.cc:2229] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:13:49.902138 22422 raft_consensus.cc:2258] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:13:49.902827 22423 raft_consensus.cc:2229] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:13:49.903517 22423 raft_consensus.cc:2258] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:13:49.904166 22424 raft_consensus.cc:2229] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:13:49.904625 22424 raft_consensus.cc:2258] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:13:49.906075 22422 ts_tablet_manager.cc:1905] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:13:49.907750 22423 ts_tablet_manager.cc:1905] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:13:49.911612 22424 ts_tablet_manager.cc:1905] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:13:49.914793 22422 ts_tablet_manager.cc:1918] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:13:49.915010 22422 log.cc:1196] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/818c9b17bdcb404683f3d2d759b0543d
I20260806 09:13:49.915707 22422 ts_tablet_manager.cc:1939] T 818c9b17bdcb404683f3d2d759b0543d P 8c6df30184d2490cbf4a5889b932b0f9: Deleting consensus metadata
I20260806 09:13:49.917814 21624 catalog_manager.cc:4841] TS 8c6df30184d2490cbf4a5889b932b0f9 (127.3.69.1:42027): tablet 818c9b17bdcb404683f3d2d759b0543d (table test-table [id=129a34b94df54849b92cbc0d03fb121a]) successfully deleted
I20260806 09:13:49.915017 22423 ts_tablet_manager.cc:1918] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:13:49.918766 22423 log.cc:1196] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/818c9b17bdcb404683f3d2d759b0543d
I20260806 09:13:49.918892 22424 ts_tablet_manager.cc:1918] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:13:49.919057 22424 log.cc:1196] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/818c9b17bdcb404683f3d2d759b0543d
I20260806 09:13:49.919378 22423 ts_tablet_manager.cc:1939] T 818c9b17bdcb404683f3d2d759b0543d P a4705b50b1e44cbcb975fa05fb10b35e: Deleting consensus metadata
I20260806 09:13:49.919672 22424 ts_tablet_manager.cc:1939] T 818c9b17bdcb404683f3d2d759b0543d P ce6c18bfeabd4fbdbb73a956c1705d19: Deleting consensus metadata
I20260806 09:13:49.921514 21628 catalog_manager.cc:4841] TS ce6c18bfeabd4fbdbb73a956c1705d19 (127.3.69.2:33813): tablet 818c9b17bdcb404683f3d2d759b0543d (table test-table [id=129a34b94df54849b92cbc0d03fb121a]) successfully deleted
I20260806 09:13:49.921653 21625 catalog_manager.cc:4841] TS a4705b50b1e44cbcb975fa05fb10b35e (127.3.69.3:39675): tablet 818c9b17bdcb404683f3d2d759b0543d (table test-table [id=129a34b94df54849b92cbc0d03fb121a]) successfully deleted
I20260806 09:13:49.926112 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:49.893440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.3.69.1:42027)
0806 09:13:49.893624 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:49.893631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:49.901288 (+ 7657us) server_negotiation.cc:407] Connection header received
0806 09:13:49.901538 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:49.901544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:49.901635 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:49.901771 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:49.903716 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.904725 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.907258 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:49.907878 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:49.908011 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:49.918728 (+ 10717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:49.918744 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:49.918762 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:49.918805 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:49.922789 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:49.923741 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:49.923750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:49.923753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:49.923917 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:49.924388 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:49.924394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:49.924397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:49.924846 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:49.925176 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:49.925515 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 09:13:49.925759 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:13:49.927019 21944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:50.084470 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 21702
I20260806 09:13:50.117403 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 21958
I20260806 09:13:50.135931 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:50.118307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.3.69.2:33813)
0806 09:13:50.118513 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:50.118520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:50.118635 (+ 115us) server_negotiation.cc:407] Connection header received
0806 09:13:50.118833 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:50.118839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:50.118924 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:50.119034 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:50.120891 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:50.122022 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:50.123945 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:50.124607 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:50.124743 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:50.128318 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:50.128339 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:50.128361 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:50.128416 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:50.132591 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:50.133472 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:50.133483 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:50.133488 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:50.133660 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:50.134092 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:50.134098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:50.134101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:50.134601 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:50.134910 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:50.135310 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 09:13:50.135544 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:13:50.136703 22158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:50.241155 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 21958
I20260806 09:13:50.269657 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 22174
I20260806 09:13:50.286659 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:50.270648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.3.69.3:39675)
0806 09:13:50.270869 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:50.270876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:50.270899 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:13:50.270977 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:50.270983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:50.271122 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:50.271254 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:50.272576 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:50.273771 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:50.275475 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:50.276210 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:50.276374 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:50.279026 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:50.279046 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:50.279070 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:50.279163 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:50.283546 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:50.284328 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:50.284345 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:50.284349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:50.284487 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:50.284905 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:50.284910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:50.284914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:50.285351 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:50.285728 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:50.286052 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 09:13:50.286318 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:13:50.287497 22346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:50.411386 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 22174
I20260806 09:13:50.439926 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 21597
I20260806 09:13:50.441267 21694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:50.554230 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 21597
I20260806 09:13:50.590838 3348 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 09:13:50.936573 3348 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 09:13:50.936864 3348 mini_ranger.cc:66] Stopping Ranger...
I20260806 09:13:51.292725 3348 mini_ranger.cc:68] Stopped Ranger
2026-08-06 09:13:51.293 UTC [20215] LOG: received smart shutdown request
2026-08-06 09:13:51.296 UTC [20215] LOG: background worker "logical replication launcher" (PID 20226) exited with exit code 1
2026-08-06 09:13:51.296 UTC [20219] LOG: shutting down
2026-08-06 09:13:51.472 UTC [20215] LOG: database system is shut down
2026-08-06T09:13:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54445 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:13:51 dist-test-slave-w4v5 krb5kdc[22447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:13:51 dist-test-slave-w4v5 krb5kdc[22447](info): set up 2 sockets
Aug 06 09:13:51 dist-test-slave-w4v5 krb5kdc[22447](info): commencing operation
krb5kdc: starting...
W20260806 09:13:52.928521 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:13:53 dist-test-slave-w4v5 krb5kdc[22447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:13:53.211462 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36633
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:36633,127.3.69.61:39067,127.3.69.60:45147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:13:53.559691 22463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:53.560038 22463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:53.560204 22463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:53.567238 22463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:13:53.567381 22463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:53.567488 22463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:53.567600 22463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:13:53.567690 22463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:13:53.577131 22463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:36633,127.3.69.61:39067,127.3.69.60:45147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:53.578416 22463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:53.579754 22463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:53.592129 22472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:53.592135 22474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:53.592170 22478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:53.593343 22463 server_base.cc:1034] running on GCE node
I20260806 09:13:53.594111 22463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:53.595584 22463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:53.596823 22463 hybrid_clock.cc:648] HybridClock initialized: now 1786007633596781 us; error 55 us; skew 500 ppm
I20260806 09:13:53.597184 22463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:53 dist-test-slave-w4v5 krb5kdc[22447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:53.604610 22463 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:13:53.606091 22463 webserver.cc:460] Webserver started at http://127.3.69.62:45819/ using document root <none> and password file <none>
I20260806 09:13:53.606832 22463 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:53.606995 22463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:13:53.607348 22463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:13:53.610386 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "7d70d00099e74b2ba69522b88d542d0b"
format_stamp: "Formatted at 2026-08-06 09:13:53 on dist-test-slave-w4v5"
server_key: "50a8231a0e5e2d631106b921fd302ea7"
server_key_iv: "beb2b8de00c1bfdc37dd8dd2bd26a509"
server_key_version: "encryptionkey@0"
I20260806 09:13:53.611317 22463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "7d70d00099e74b2ba69522b88d542d0b"
format_stamp: "Formatted at 2026-08-06 09:13:53 on dist-test-slave-w4v5"
server_key: "50a8231a0e5e2d631106b921fd302ea7"
server_key_iv: "beb2b8de00c1bfdc37dd8dd2bd26a509"
server_key_version: "encryptionkey@0"
I20260806 09:13:53.616497 22463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:13:53.619959 22490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:53.621294 22463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:13:53.621479 22463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "7d70d00099e74b2ba69522b88d542d0b"
format_stamp: "Formatted at 2026-08-06 09:13:53 on dist-test-slave-w4v5"
server_key: "50a8231a0e5e2d631106b921fd302ea7"
server_key_iv: "beb2b8de00c1bfdc37dd8dd2bd26a509"
server_key_version: "encryptionkey@0"
I20260806 09:13:53.621678 22463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:53.665838 22463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:53.670269 22463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:53.670603 22463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:53.691103 22463 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:36633
I20260806 09:13:53.691123 22564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:36633 every 8 connection(s)
I20260806 09:13:53.692586 22463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:13:53.696552 22565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:13:53.701375 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 22463
I20260806 09:13:53.701632 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:13:53.702560 3348 external_mini_cluster.cc:1428] Setting key 7a820930247407493b2c930bd71a048d
I20260806 09:13:53.704449 22565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:13:53 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:53.728314 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.708360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36633 (local address 127.0.0.1:39210)
0806 09:13:53.709253 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.709298 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:53.711381 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:53.712185 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:53.712191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:53.712899 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:53.713237 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:53.713257 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:53.714811 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:53.714817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:53.715948 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:53.715961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:53.716883 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:53.716889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:53.717078 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:53.717761 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:53.717783 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:53.720575 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:53.723905 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:53.723920 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:53.723935 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:53.724624 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:53.725043 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:53.725051 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:53.725057 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:53.725287 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:53.726225 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:53.726235 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:53.726567 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 09:13:53.726793 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 09:13:53.727253 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":2,"thread_start_us":411,"threads_started":1}
I20260806 09:13:53.728363 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.708364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.3.69.62:36633)
0806 09:13:53.710098 (+ 1734us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:53.710106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:53.710161 (+ 55us) server_negotiation.cc:407] Connection header received
0806 09:13:53.711745 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:53.711750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:53.711957 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:53.712158 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:53.713441 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:53.714636 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:53.716092 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:53.716632 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:53.716836 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:53.720810 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:53.720818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:53.720828 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:53.720857 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:53.723759 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:53.724745 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:53.724751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:53.724753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:53.724898 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:53.725439 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:53.725443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:53.725445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:53.726052 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:53.726302 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:53.727014 (+ 712us) server_negotiation.cc:299] Negotiation successful
0806 09:13:53.727444 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":1521,"thread_start_us":1274,"threads_started":1}
WARNING: no policy specified for kudu/127.3.69.61@KRBTEST.COM; defaulting to no policy
I20260806 09:13:53.731534 22565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:13:53.732630 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.732203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41830)
0806 09:13:53.732357 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.732467 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:13:53.732893 22493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.61@KRBTEST.COM" created.
I20260806 09:13:53.734668 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.734283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41836)
0806 09:13:53.734431 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.734548 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 09:13:53.734949 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39067: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab.
Entry for principal kudu/127.3.69.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab.
I20260806 09:13:53.766995 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.61:39067
--webserver_interface=127.3.69.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab
--principal=kudu/127.3.69.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:36633,127.3.69.61:39067,127.3.69.60:45147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:13:53.784400 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } attempt: 1
I20260806 09:13:53.785887 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.785220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41838)
0806 09:13:53.785499 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.785641 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 09:13:53.787200 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.786650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41846)
0806 09:13:53.786854 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.786965 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 09:13:53.787465 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39067: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
I20260806 09:13:53.855786 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } attempt: 2
I20260806 09:13:53.857184 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.856566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41848)
0806 09:13:53.856858 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.856983 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 09:13:53.858471 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.857951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41850)
0806 09:13:53.858153 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.858276 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:13:53.858716 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39067: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
I20260806 09:13:53.950028 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } attempt: 3
I20260806 09:13:53.951373 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.950791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41862)
0806 09:13:53.951033 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.951193 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 09:13:53.952445 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:53.952028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41874)
0806 09:13:53.952184 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:53.952271 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:13:53.952732 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39067: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
I20260806 09:13:54.096041 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } attempt: 4
I20260806 09:13:54.097467 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.096806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41876)
0806 09:13:54.097174 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.097294 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260806 09:13:54.098616 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.098174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41884)
0806 09:13:54.098340 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.098455 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 09:13:54.098820 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39067: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
W20260806 09:13:54.111496 22579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:54.111801 22579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:54.111894 22579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:54.119288 22579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:13:54.119400 22579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:54.119496 22579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:54.119601 22579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:13:54.119709 22579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:13:54.129801 22579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:36633,127.3.69.61:39067,127.3.69.60:45147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab
--principal=kudu/127.3.69.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.61:39067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.69.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:54.131255 22579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:54.132550 22579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:54.144831 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:54.144726 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:54.144805 22596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:54.145483 22579 server_base.cc:1034] running on GCE node
I20260806 09:13:54.146170 22579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:54.147475 22579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:54.148703 22579 hybrid_clock.cc:648] HybridClock initialized: now 1786007634148649 us; error 40 us; skew 500 ppm
I20260806 09:13:54.149046 22579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:54.152408 22579 init.cc:377] Logged in from keytab as kudu/127.3.69.61@KRBTEST.COM (short username kudu)
I20260806 09:13:54.153894 22579 webserver.cc:460] Webserver started at http://127.3.69.61:45437/ using document root <none> and password file <none>
I20260806 09:13:54.154568 22579 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:54.154721 22579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:13:54.155035 22579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:13:54.159060 22579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/data/instance:
uuid: "f353bccf28d74fd9b07c7ba95d7bb941"
format_stamp: "Formatted at 2026-08-06 09:13:54 on dist-test-slave-w4v5"
server_key: "4642c790ec3081cee40f0867a5ac5a03"
server_key_iv: "7469b4fd9313d26406269f516d1cf3cc"
server_key_version: "encryptionkey@0"
I20260806 09:13:54.159981 22579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/wal/instance:
uuid: "f353bccf28d74fd9b07c7ba95d7bb941"
format_stamp: "Formatted at 2026-08-06 09:13:54 on dist-test-slave-w4v5"
server_key: "4642c790ec3081cee40f0867a5ac5a03"
server_key_iv: "7469b4fd9313d26406269f516d1cf3cc"
server_key_version: "encryptionkey@0"
I20260806 09:13:54.165162 22579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 09:13:54.168604 22606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:54.169839 22579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:13:54.170030 22579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/wal
uuid: "f353bccf28d74fd9b07c7ba95d7bb941"
format_stamp: "Formatted at 2026-08-06 09:13:54 on dist-test-slave-w4v5"
server_key: "4642c790ec3081cee40f0867a5ac5a03"
server_key_iv: "7469b4fd9313d26406269f516d1cf3cc"
server_key_version: "encryptionkey@0"
I20260806 09:13:54.170241 22579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:54.190521 22579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:54.195075 22579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:54.195447 22579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:54.215382 22579 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.61:39067
I20260806 09:13:54.215385 22685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.61:39067 every 8 connection(s)
I20260806 09:13:54.216979 22579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
I20260806 09:13:54.220762 22687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:13:54.223546 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 22579
I20260806 09:13:54.223776 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-1/wal/instance
I20260806 09:13:54.224380 3348 external_mini_cluster.cc:1428] Setting key 6c68edbac61aabe4ce25224d8f867029
I20260806 09:13:54.230885 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.69.60@KRBTEST.COM; defaulting to no policy
I20260806 09:13:54.255658 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.234568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.3.69.62:36633)
0806 09:13:54.235192 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.235201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.236349 (+ 1148us) server_negotiation.cc:407] Connection header received
0806 09:13:54.238374 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.238379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.238452 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.238546 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.240867 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.241587 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.244281 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.244726 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.244841 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.249298 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.249302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.249306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.249351 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.252391 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.253187 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.253191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.253194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.253314 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.253776 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.253780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.253782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.254061 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.254246 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.255016 (+ 770us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.255277 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":298,"threads_started":1}
I20260806 09:13:54.256661 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.235268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36633 (local address 127.0.0.1:39212)
0806 09:13:54.236139 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.236198 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:54.238026 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:54.238827 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:54.238841 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:54.239731 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:54.240652 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.240682 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.241775 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.241783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.244032 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.244051 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.244981 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.244990 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.245328 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.246105 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:54.246152 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:54.249036 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:54.252543 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.252552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.252569 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.253036 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.253432 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.253437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.253441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.253641 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.254204 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:54.254212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:54.254606 (+ 394us) client_negotiation.cc:769] Sending connection context
0806 09:13:54.254918 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 09:13:54.255493 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":391,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.60@KRBTEST.COM" created.
I20260806 09:13:54.258594 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:13:54.273146 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.259308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41886)
0806 09:13:54.259516 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.259531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:54.259616 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:54.260657 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:54.260661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:54.261110 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:54.261272 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.261281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.262652 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.262660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.263564 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.263575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.264412 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.264418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.264629 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.265234 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:54.265250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:54.267315 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:54.270552 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.270556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.270560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.270937 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.271346 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.271352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.271355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.271499 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.272092 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:54.272097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:54.272286 (+ 189us) client_negotiation.cc:769] Sending connection context
0806 09:13:54.272430 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 09:13:54.272705 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 09:13:54.273226 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.259624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.3.69.61:39067)
0806 09:13:54.260266 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.260272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.260297 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:13:54.260338 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.260344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.260510 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.260672 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.261433 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.262498 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.263705 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.264249 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.264396 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.267533 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.267545 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.267558 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.267591 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.270375 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.271068 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.271098 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.271101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.271236 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.271603 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.271610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.271613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.271971 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.272203 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.272575 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.272813 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":499,"thread_start_us":356,"threads_started":1}
I20260806 09:13:54.275974 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:13:54.277050 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.276588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55536)
0806 09:13:54.276782 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.276895 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 09:13:54.277289 22610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
I20260806 09:13:54.278923 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.278587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55552)
0806 09:13:54.278748 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.278822 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:13:54.279189 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab.
Entry for principal kudu/127.3.69.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab.
I20260806 09:13:54.292376 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.60:45147
--webserver_interface=127.3.69.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab
--principal=kudu/127.3.69.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:36633,127.3.69.61:39067,127.3.69.60:45147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:13:54.328482 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 1
I20260806 09:13:54.329770 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.329181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55568)
0806 09:13:54.329431 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.329578 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 09:13:54.330878 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.330464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55580)
0806 09:13:54.330640 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.330744 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 09:13:54.331161 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
I20260806 09:13:54.398109 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } attempt: 5
I20260806 09:13:54.399391 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 2
I20260806 09:13:54.400411 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.399906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55590)
0806 09:13:54.400104 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.400210 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 09:13:54.401612 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.401176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55604)
0806 09:13:54.401357 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.401444 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 09:13:54.401834 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:13:54.412484 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.399175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.3.69.61:39067)
0806 09:13:54.399387 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.399393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.399420 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:13:54.399497 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.399501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.399613 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.399725 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.401029 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.402085 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.403302 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.403754 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.403889 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.407161 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.407166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.407169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.407200 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.409847 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.410530 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.410538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.410541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.410675 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.411055 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.411061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.411064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.411387 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.411547 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.411939 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.412168 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:13:54.412495 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.398900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41902)
0806 09:13:54.399164 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.399187 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:54.399360 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:54.399896 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:54.399900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:54.400473 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:54.400863 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.400877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.402235 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.402247 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.403164 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.403176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.403914 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.403921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.404139 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.404804 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:54.404822 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:54.406917 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:54.410003 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.410008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.410012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.410388 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.410794 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.410799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.410803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.410942 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.411513 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:54.411517 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:54.411678 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 09:13:54.411825 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 09:13:54.412139 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 09:13:54.413542 22565 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:13:54.414553 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.414133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55608)
0806 09:13:54.414335 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.414417 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 09:13:54.415511 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.415194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55624)
0806 09:13:54.415335 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.415413 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 09:13:54.415720 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
I20260806 09:13:54.467002 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 1
I20260806 09:13:54.468376 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.467781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55636)
0806 09:13:54.468023 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.468157 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:13:54.469435 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.469051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55644)
0806 09:13:54.469210 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.469297 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:13:54.469650 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
I20260806 09:13:54.493063 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 3
I20260806 09:13:54.494485 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.493940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55654)
0806 09:13:54.494221 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.494320 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 09:13:54.495793 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.495349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55664)
0806 09:13:54.495591 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.495675 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260806 09:13:54.496054 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
I20260806 09:13:54.537882 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 2
I20260806 09:13:54.539238 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.538574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55670)
0806 09:13:54.538856 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.538977 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 09:13:54.540462 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.540035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55674)
0806 09:13:54.540216 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.540313 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:13:54.540715 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
I20260806 09:13:54.639287 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 4
I20260806 09:13:54.640437 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.639945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55680)
0806 09:13:54.640202 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.640286 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 09:13:54.641474 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.641140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55686)
0806 09:13:54.641312 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.641375 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:13:54.641747 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
W20260806 09:13:54.643030 22702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:54.643401 22702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:54.643561 22702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 09:13:54.646952 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 3
I20260806 09:13:54.648018 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.647522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55696)
0806 09:13:54.647731 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.647833 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 09:13:54.649078 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.648647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55700)
0806 09:13:54.648842 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.648932 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:13:54.649287 22565 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:45147: Network error: Client connection negotiation failed: client connection to 127.3.69.60:45147: connect: Connection refused (error 111)
W20260806 09:13:54.651904 22702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:13:54.652019 22702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:54.652100 22702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:54.652230 22702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:13:54.652312 22702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:13:54.662225 22702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:36633,127.3.69.61:39067,127.3.69.60:45147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab
--principal=kudu/127.3.69.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.60:45147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.69.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:54.663617 22702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:54.664937 22702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:54.678145 22716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:54.678054 22709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:54.678172 22712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:54.679028 22702 server_base.cc:1034] running on GCE node
I20260806 09:13:54.679988 22702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:54.681522 22702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:54.682777 22702 hybrid_clock.cc:648] HybridClock initialized: now 1786007634682715 us; error 54 us; skew 500 ppm
I20260806 09:13:54.683238 22702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:54.686630 22702 init.cc:377] Logged in from keytab as kudu/127.3.69.60@KRBTEST.COM (short username kudu)
I20260806 09:13:54.688195 22702 webserver.cc:460] Webserver started at http://127.3.69.60:35407/ using document root <none> and password file <none>
I20260806 09:13:54.688926 22702 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:54.689103 22702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:13:54.689450 22702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:13:54.692830 22702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/data/instance:
uuid: "63431ea0fdf64a29a88d4156cbf2a010"
format_stamp: "Formatted at 2026-08-06 09:13:54 on dist-test-slave-w4v5"
server_key: "1e1a5ee241c5f7d61b55964c6e95e82a"
server_key_iv: "4d6be810d40545b2ef05ffa720628e42"
server_key_version: "encryptionkey@0"
I20260806 09:13:54.693748 22702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/wal/instance:
uuid: "63431ea0fdf64a29a88d4156cbf2a010"
format_stamp: "Formatted at 2026-08-06 09:13:54 on dist-test-slave-w4v5"
server_key: "1e1a5ee241c5f7d61b55964c6e95e82a"
server_key_iv: "4d6be810d40545b2ef05ffa720628e42"
server_key_version: "encryptionkey@0"
I20260806 09:13:54.699390 22702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 09:13:54.703015 22723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:54.704455 22702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:13:54.704664 22702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/wal
uuid: "63431ea0fdf64a29a88d4156cbf2a010"
format_stamp: "Formatted at 2026-08-06 09:13:54 on dist-test-slave-w4v5"
server_key: "1e1a5ee241c5f7d61b55964c6e95e82a"
server_key_iv: "4d6be810d40545b2ef05ffa720628e42"
server_key_version: "encryptionkey@0"
I20260806 09:13:54.704872 22702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:54.717336 22702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:54.721264 22702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:54.721555 22702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:54.741392 22702 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.60:45147
I20260806 09:13:54.741441 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.60:45147 every 8 connection(s)
I20260806 09:13:54.742784 22702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
I20260806 09:13:54.746695 22801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:13:54.750809 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 22702
I20260806 09:13:54.751024 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786007418649204-3348-0/minicluster-data/master-2/wal/instance
I20260806 09:13:54.751609 3348 external_mini_cluster.cc:1428] Setting key 343074c86befddfc317fbc6644bfc200
I20260806 09:13:54.754393 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:54.778541 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.753447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.3.69.60:45147)
0806 09:13:54.754241 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.754251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.754292 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:13:54.756470 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.756492 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.756953 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.757406 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.759927 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.762440 (+ 2513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.764727 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.765534 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.765780 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.770800 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.770814 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.770830 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.770880 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.774430 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.775227 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.775236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.775247 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.775410 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.775810 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.775816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.775819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.776325 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.776663 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.777084 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.777555 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":384,"threads_started":1}
I20260806 09:13:54.780197 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.758785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.3.69.62:36633)
0806 09:13:54.763984 (+ 5199us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.763990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.764009 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:13:54.764095 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.764100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.764179 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.764301 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.766061 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.767676 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.769587 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.770122 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.770261 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.774325 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.774330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.774333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.774366 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.777125 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.777977 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.777985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.777988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.778128 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.778502 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.778509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.778512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.778839 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.779003 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.779561 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.779798 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5046,"thread_start_us":4900,"threads_started":1}
I20260806 09:13:54.780226 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.758566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36633 (local address 127.0.0.1:39226)
0806 09:13:54.763296 (+ 4730us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.763349 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:54.763496 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:54.764533 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:54.764540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:54.765418 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:54.765857 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.765872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.767865 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.767872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.769442 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.769455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.770360 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.770368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.770542 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.771211 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:54.771234 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:54.774109 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:54.777273 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.777295 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.777302 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.777851 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.778242 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.778248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.778251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.778392 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.779000 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:54.779007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:54.779263 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 09:13:54.779439 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 09:13:54.779750 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4460,"thread_start_us":5714,"threads_started":1}
I20260806 09:13:54.781695 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:54.795107 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.782321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:41904)
0806 09:13:54.782487 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.782505 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:54.782595 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:54.782938 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:54.782941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:54.783311 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:54.783445 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.783452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.784534 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.784540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.785460 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.785471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.786171 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.786177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.786385 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.786970 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:54.786995 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:54.789001 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:54.792488 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.792496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.792501 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.793056 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.793443 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.793450 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.793453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.793637 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.794122 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:54.794127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:54.794299 (+ 172us) client_negotiation.cc:769] Sending connection context
0806 09:13:54.794445 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 09:13:54.794721 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 09:13:54.795179 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.782442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.3.69.61:39067)
0806 09:13:54.782626 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.782633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.782653 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:13:54.782718 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.782725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.782809 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.782924 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.783603 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.784373 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.785600 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.786018 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.786164 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.789211 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.789218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.789223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.789266 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.792339 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.793174 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.793177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.793180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.793325 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.793748 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.793751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.793753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.793993 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.794160 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.794538 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.794785 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 09:13:54.796110 22801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:13:54.796895 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.781706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.3.69.62:36633)
0806 09:13:54.781871 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.781877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.781895 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:13:54.782017 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.782022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.782125 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.782207 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.783796 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.784447 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.785751 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.786161 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.786295 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.790721 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.790726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.790729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.790759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.793973 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.794681 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.794688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.794691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.794819 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.795347 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.795352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.795355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.795708 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.796003 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.796307 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.796539 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:13:54.811532 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.797127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.3.69.60:45147)
0806 09:13:54.797269 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.797273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.797286 (+ 13us) server_negotiation.cc:407] Connection header received
0806 09:13:54.797322 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.797325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.797376 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.797450 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.798349 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.799381 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.801057 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.801764 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.801880 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.805526 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.805531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.805534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.805563 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.808909 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.809719 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.809723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.809725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.809835 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.810208 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.810215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.810218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.810515 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.810666 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.810999 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.811279 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260806 09:13:54.811939 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.796807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55728)
0806 09:13:54.797002 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.797023 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:54.797123 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:54.797498 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:54.797503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:54.797992 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:54.798175 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.798187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.799515 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.799524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.800922 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.800934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.801909 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.801915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.802138 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.802687 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:54.802710 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:54.805333 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:54.809042 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.809047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.809051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.809591 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.809946 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.809951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.809954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.810099 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.810631 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:54.810636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:54.810769 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 09:13:54.810921 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 09:13:54.811479 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":193}
I20260806 09:13:54.815454 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.798769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.3.69.61:39067)
0806 09:13:54.798936 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.798942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.798961 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:13:54.799029 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.799034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.799127 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.799220 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.800735 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.801728 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.803691 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.804252 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.804405 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.809135 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.809141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.809145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.809188 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.812483 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.813358 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.813362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.813365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.813536 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.813943 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.813950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.813954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.814306 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.814589 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.814829 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.815120 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 09:13:54.818014 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010: Bootstrap starting.
I20260806 09:13:54.820902 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:13:54.821928 22801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010: Neither blocks nor log segments found. Creating new log.
I20260806 09:13:54.823333 22801 log.cc:824] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010: Log is configured to *not* fsync() on all Append() calls
I20260806 09:13:54.826488 22565 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 4
I20260806 09:13:54.826683 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010: No bootstrap required, opened a new log
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
I20260806 09:13:54.838218 22801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:54.838732 22801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:13:54.838904 22801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63431ea0fdf64a29a88d4156cbf2a010, State: Initialized, Role: FOLLOWER
I20260806 09:13:54.839821 22801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:54.849399 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.827143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55738)
0806 09:13:54.827416 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.827435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:54.827547 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:54.828071 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:54.828077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:54.828634 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:54.828908 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.828922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.830560 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.830568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.832151 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.832167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.833440 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.833446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.833643 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.834355 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:54.834380 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:54.837350 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:54.845264 (+ 7914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.845273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.845279 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.846443 (+ 1164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.846939 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.846945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.846949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.847168 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.848023 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:54.848029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:54.848230 (+ 201us) client_negotiation.cc:769] Sending connection context
0806 09:13:54.848419 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 09:13:54.848813 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":24}
I20260806 09:13:54.850466 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.827372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.3.69.60:45147)
0806 09:13:54.827609 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.827615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.827637 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:13:54.827696 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.827702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.827772 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.827875 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.829091 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.830335 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.832326 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.833073 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.833198 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.837559 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.837564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.837581 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.837616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.844675 (+ 7059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.846596 (+ 1921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.846602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.846607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.846799 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.847310 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.847315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.847319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.847775 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.847953 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.849800 (+ 1847us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.850100 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:13:54.854611 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.827406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.3.69.62:36633)
0806 09:13:54.827543 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.827547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.828220 (+ 673us) server_negotiation.cc:407] Connection header received
0806 09:13:54.828483 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.828488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.828562 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.828632 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.831476 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.832970 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.835750 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.836665 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.836787 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.841850 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.841857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.841863 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.841905 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.845917 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.851764 (+ 5847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.851771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.851776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.851952 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.852531 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.852537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.852542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.852953 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.853245 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.853922 (+ 677us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.854196 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":51}
I20260806 09:13:54.859725 22565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: Bootstrap starting.
I20260806 09:13:54.865522 22565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: Neither blocks nor log segments found. Creating new log.
I20260806 09:13:54.866087 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.828642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.3.69.61:39067)
0806 09:13:54.828822 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.828828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.840155 (+ 11327us) server_negotiation.cc:407] Connection header received
0806 09:13:54.841127 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.841133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.841234 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.841311 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.844977 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.846107 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.850101 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.850858 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.850961 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.857173 (+ 6212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.857182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.857188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.857235 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.860927 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.862481 (+ 1554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.862488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.862493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.862664 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.863582 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.863588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.863591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.864016 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.864317 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.865422 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.865694 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:13:54.868770 22565 log.cc:824] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:13:54.870414 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:54.871249 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:13:54.872260 22801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:13:54.874415 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.840863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.3.69.60:45147)
0806 09:13:54.851637 (+ 10774us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.851655 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.855880 (+ 4225us) server_negotiation.cc:407] Connection header received
0806 09:13:54.855952 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.855961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.856053 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.856154 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.857922 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.859119 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.861152 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.861907 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.862034 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.865689 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.865697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.865701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.865742 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.870205 (+ 4463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.871183 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.871189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.871193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.871437 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.871978 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.871983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.871986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.872399 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.872669 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.873906 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.874170 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10656,"thread_start_us":25951,"threads_started":1}
I20260806 09:13:54.877353 22565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: No bootstrap required, opened a new log
W20260806 09:13:54.885924 22838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:13:54.886112 22838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:13:54.888895 22565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:54.889330 22565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:13:54.889518 22565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d70d00099e74b2ba69522b88d542d0b, State: Initialized, Role: FOLLOWER
I20260806 09:13:54.890182 22565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:54.891247 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:54.892556 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:13:54.892591 22565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:13:54.881275 22702 master_runner.cc:186] Error getting master registration for 127.3.69.62:36633: OK, instance_id { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" instance_seqno: 1786007633685167 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:13:54.893468 22702 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 09:13:54.903930 22859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:13:54.904196 22859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:13:54.904249 22463 master_runner.cc:186] Error getting master registration for 127.3.69.61:39067: OK, instance_id { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" instance_seqno: 1786007634209595 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:13:54.918151 22463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 09:13:54.923648 22500 tablet.cc:2369] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:13:54.940994 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } attempt: 5
Aug 06 09:13:54 dist-test-slave-w4v5 krb5kdc[22447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
I20260806 09:13:54.958827 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.941627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:45147 (local address 127.0.0.1:55760)
0806 09:13:54.941888 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:54.941911 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:13:54.942034 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:13:54.942420 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:13:54.942425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:13:54.942981 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:13:54.943234 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.943247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.944783 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.944792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.946247 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:13:54.946262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.947540 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.947547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:13:54.947849 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.948545 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:13:54.948570 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:13:54.951495 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:13:54.955918 (+ 4423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.955926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.955930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.956466 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.956927 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:13:54.956933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:13:54.956937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:13:54.957112 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:13:54.957756 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:13:54.957761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:13:54.957928 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 09:13:54.958092 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 09:13:54.958386 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 09:13:54.959193 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:54.941703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.3.69.60:45147)
0806 09:13:54.941891 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:54.941901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:54.941968 (+ 67us) server_negotiation.cc:407] Connection header received
0806 09:13:54.942166 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:54.942172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:54.942252 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:54.942363 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:54.943400 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.944597 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.946412 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:54.947175 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:54.947307 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:54.951717 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:54.951724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:54.951729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:54.951779 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:54.955773 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.956596 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.956601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.956605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.956800 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:54.957241 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:54.957246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:54.957249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:54.957609 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:54.957823 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:54.958213 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 09:13:54.958461 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:13:54.965243 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: Bootstrap starting.
I20260806 09:13:54.969488 22687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: Neither blocks nor log segments found. Creating new log.
I20260806 09:13:54.970873 22687 log.cc:824] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: Log is configured to *not* fsync() on all Append() calls
W20260806 09:13:54.973657 22730 tablet.cc:2369] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:13:54.974164 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: No bootstrap required, opened a new log
I20260806 09:13:54.985327 22687 raft_consensus.cc:348] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:54.985743 22687 raft_consensus.cc:374] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:13:54.985864 22687 raft_consensus.cc:729] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f353bccf28d74fd9b07c7ba95d7bb941, State: Initialized, Role: FOLLOWER
I20260806 09:13:54.986510 22687 consensus_queue.cc:260] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:54.987584 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:54.988073 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:13:54.988904 22687 sys_catalog.cc:564] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:13:55.007125 22579 master_runner.cc:263] No leader master found from master f353bccf28d74fd9b07c7ba95d7bb941
I20260806 09:13:55.007284 22579 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 09:13:55.013999 22879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:13:55.014196 22879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:13:55.145922 22820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:13:55.146273 22820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:55.148203 22820 leader_election.cc:288] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d70d00099e74b2ba69522b88d542d0b (127.3.69.62:36633), f353bccf28d74fd9b07c7ba95d7bb941 (127.3.69.61:39067)
I20260806 09:13:55.149513 22529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63431ea0fdf64a29a88d4156cbf2a010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70d00099e74b2ba69522b88d542d0b" is_pre_election: true
I20260806 09:13:55.150120 22529 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63431ea0fdf64a29a88d4156cbf2a010 in term 0.
I20260806 09:13:55.150990 22725 leader_election.cc:302] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63431ea0fdf64a29a88d4156cbf2a010, 7d70d00099e74b2ba69522b88d542d0b; no voters:
I20260806 09:13:55.152273 22658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63431ea0fdf64a29a88d4156cbf2a010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" is_pre_election: true
I20260806 09:13:55.152910 22658 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63431ea0fdf64a29a88d4156cbf2a010 in term 0.
I20260806 09:13:55.153469 22820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:13:55.153650 22820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:13:55.153759 22820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:13:55.157935 22820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:55.162875 22529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63431ea0fdf64a29a88d4156cbf2a010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d70d00099e74b2ba69522b88d542d0b"
I20260806 09:13:55.163143 22529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:13:55.166679 22529 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63431ea0fdf64a29a88d4156cbf2a010 in term 1.
I20260806 09:13:55.167548 22725 leader_election.cc:302] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63431ea0fdf64a29a88d4156cbf2a010, 7d70d00099e74b2ba69522b88d542d0b; no voters:
I20260806 09:13:55.170015 22820 leader_election.cc:288] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [CANDIDATE]: Term 1 election: Requested vote from peers 7d70d00099e74b2ba69522b88d542d0b (127.3.69.62:36633), f353bccf28d74fd9b07c7ba95d7bb941 (127.3.69.61:39067)
I20260806 09:13:55.170569 22820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:13:55.171022 22820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [term 1 LEADER]: Becoming Leader. State: Replica: 63431ea0fdf64a29a88d4156cbf2a010, State: Running, Role: LEADER
I20260806 09:13:55.175974 22820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } }
I20260806 09:13:55.179915 22658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63431ea0fdf64a29a88d4156cbf2a010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f353bccf28d74fd9b07c7ba95d7bb941"
I20260806 09:13:55.180145 22658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:13:55.183837 22658 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63431ea0fdf64a29a88d4156cbf2a010 in term 1.
I20260806 09:13:55.191977 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63431ea0fdf64a29a88d4156cbf2a010. Latest consensus state: current_term: 1 leader_uuid: "63431ea0fdf64a29a88d4156cbf2a010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:55.192451 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: This master's current role is: LEADER
I20260806 09:13:55.195694 22885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260806 09:13:55.198094 22617 tablet.cc:2369] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:13:55.201184 22885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:13:55.225361 22658 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [term 1 FOLLOWER]: Refusing update from remote peer 63431ea0fdf64a29a88d4156cbf2a010: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:13:55.226810 22529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [term 1 FOLLOWER]: Refusing update from remote peer 63431ea0fdf64a29a88d4156cbf2a010: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:13:55.228554 22883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:13:55.228888 22820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:13:55.263505 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63431ea0fdf64a29a88d4156cbf2a010. Latest consensus state: current_term: 1 leader_uuid: "63431ea0fdf64a29a88d4156cbf2a010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:55.263862 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:13:55.272552 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63431ea0fdf64a29a88d4156cbf2a010. Latest consensus state: current_term: 1 leader_uuid: "63431ea0fdf64a29a88d4156cbf2a010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:55.272912 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:13:55.291863 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63431ea0fdf64a29a88d4156cbf2a010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:55.292184 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:13:55.296124 22885 catalog_manager.cc:1347] Generated new cluster ID: 9de044c30ff34473a8c368d7c6aa0a88
I20260806 09:13:55.296255 22885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:13:55.296485 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63431ea0fdf64a29a88d4156cbf2a010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:55.296862 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: This master's current role is: LEADER
I20260806 09:13:55.297422 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63431ea0fdf64a29a88d4156cbf2a010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:55.297715 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010 [sys.catalog]: This master's current role is: LEADER
I20260806 09:13:55.298058 22890 mvcc.cc:205] Tried to move back new op lower bound from 7315487273851338752 to 7315487273690640384. Current Snapshot: MvccSnapshot[applied={T|T < 7315487273851338752}]
I20260806 09:13:55.298897 22892 mvcc.cc:205] Tried to move back new op lower bound from 7315487273851338752 to 7315487273690640384. Current Snapshot: MvccSnapshot[applied={T|T < 7315487273851338752 or (T in {7315487273851338752})}]
I20260806 09:13:55.300127 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63431ea0fdf64a29a88d4156cbf2a010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d70d00099e74b2ba69522b88d542d0b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36633 } } peers { permanent_uuid: "f353bccf28d74fd9b07c7ba95d7bb941" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39067 } } peers { permanent_uuid: "63431ea0fdf64a29a88d4156cbf2a010" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 45147 } } }
I20260806 09:13:55.300393 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:13:55.324283 22885 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:13:55.330003 22885 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:13:55.362784 22885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63431ea0fdf64a29a88d4156cbf2a010: Generated new TSK 0
I20260806 09:13:55.364145 22885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:13:55.906572 22859 catalog_manager.cc:1259] Loaded cluster ID: 9de044c30ff34473a8c368d7c6aa0a88
I20260806 09:13:55.906759 22859 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: loading cluster ID for follower catalog manager: success
I20260806 09:13:55.909411 22859 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: acquiring CA information for follower catalog manager: success
I20260806 09:13:55.911186 22859 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d70d00099e74b2ba69522b88d542d0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:13:56.016383 22879 catalog_manager.cc:1259] Loaded cluster ID: 9de044c30ff34473a8c368d7c6aa0a88
I20260806 09:13:56.016494 22879 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: loading cluster ID for follower catalog manager: success
I20260806 09:13:56.019023 22879 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: acquiring CA information for follower catalog manager: success
I20260806 09:13:56.020941 22879 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f353bccf28d74fd9b07c7ba95d7bb941: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:13:56.479105 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 22463
I20260806 09:13:56.480520 22557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:56.592001 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 22463
I20260806 09:13:56.624187 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 22579
I20260806 09:13:56.625367 22680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:56.734498 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 22579
I20260806 09:13:56.764982 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 22702
I20260806 09:13:56.766252 22793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:56.880151 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:56.879196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36633 (local address 127.0.0.1:42132)
0806 09:13:56.879690 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:56.879822 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":250,"threads_started":1}
W20260806 09:13:56.880488 22725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.62:36633: connect: Connection refused (error 111)
I20260806 09:13:56.880638 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 22702
I20260806 09:13:56.880688 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:56.879809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39067 (local address 127.0.0.1:57590)
0806 09:13:56.880219 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0806 09:13:56.880383 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":231,"threads_started":1}
2026-08-06T09:13:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5278 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:13:57 dist-test-slave-w4v5 krb5kdc[22927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:13:57 dist-test-slave-w4v5 krb5kdc[22927](info): set up 2 sockets
Aug 06 09:13:57 dist-test-slave-w4v5 krb5kdc[22927](info): commencing operation
krb5kdc: starting...
W20260806 09:13:58.206827 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:13:58 dist-test-slave-w4v5 krb5kdc[22927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:13:58.484452 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40033
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:44103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:13:58.847440 22948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:13:58.847756 22948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:13:58.847851 22948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:13:58.855257 22948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:13:58.855363 22948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:13:58.855410 22948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:13:58.855477 22948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:13:58.855525 22948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:13:58.865408 22948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:44103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:40033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:13:58.866670 22948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:13:58.867933 22948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:13:58.879957 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:58.880000 22961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:13:58.880151 22955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:13:58.881373 22948 server_base.cc:1034] running on GCE node
I20260806 09:13:58.882140 22948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:13:58.883550 22948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:13:58.884812 22948 hybrid_clock.cc:648] HybridClock initialized: now 1786007638884759 us; error 55 us; skew 500 ppm
I20260806 09:13:58.885210 22948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:13:58 dist-test-slave-w4v5 krb5kdc[22927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:13:58.891728 22948 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:13:58.893239 22948 webserver.cc:460] Webserver started at http://127.3.69.62:38821/ using document root <none> and password file <none>
I20260806 09:13:58.893940 22948 fs_manager.cc:359] Metadata directory not provided
I20260806 09:13:58.894083 22948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:13:58.894402 22948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:13:58.897475 22948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "86eaa1028ea14a0db7cb20235501ecd8"
format_stamp: "Formatted at 2026-08-06 09:13:58 on dist-test-slave-w4v5"
server_key: "e96d146b073add7c3f0984029ab1ba2e"
server_key_iv: "85dfbf1c72fda59036095710cbcd9f31"
server_key_version: "encryptionkey@0"
I20260806 09:13:58.898247 22948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "86eaa1028ea14a0db7cb20235501ecd8"
format_stamp: "Formatted at 2026-08-06 09:13:58 on dist-test-slave-w4v5"
server_key: "e96d146b073add7c3f0984029ab1ba2e"
server_key_iv: "85dfbf1c72fda59036095710cbcd9f31"
server_key_version: "encryptionkey@0"
I20260806 09:13:58.903543 22948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 09:13:58.907001 22971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:13:58.908419 22948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:13:58.908615 22948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "86eaa1028ea14a0db7cb20235501ecd8"
format_stamp: "Formatted at 2026-08-06 09:13:58 on dist-test-slave-w4v5"
server_key: "e96d146b073add7c3f0984029ab1ba2e"
server_key_iv: "85dfbf1c72fda59036095710cbcd9f31"
server_key_version: "encryptionkey@0"
I20260806 09:13:58.908825 22948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:13:58.925808 22948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:13:58.929797 22948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:13:58.930091 22948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:13:58.949357 22948 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:40033
I20260806 09:13:58.949388 23051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:40033 every 8 connection(s)
I20260806 09:13:58.950845 22948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:13:58.954682 23052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:13:58.960569 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 22948
I20260806 09:13:58.960812 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:13:58.961596 3348 external_mini_cluster.cc:1428] Setting key c3473e412d10f7561523ae28b09b9004
I20260806 09:13:58.965427 23052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8: Bootstrap starting.
I20260806 09:13:58.969322 23052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8: Neither blocks nor log segments found. Creating new log.
I20260806 09:13:58.970371 23052 log.cc:824] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:13:58.973515 23052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8: No bootstrap required, opened a new log
Aug 06 09:13:58 dist-test-slave-w4v5 krb5kdc[22927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:13:58.984452 23052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eaa1028ea14a0db7cb20235501ecd8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40033 } }
I20260806 09:13:58.984938 23052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:13:58.985092 23052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86eaa1028ea14a0db7cb20235501ecd8, State: Initialized, Role: FOLLOWER
I20260806 09:13:58.985812 23052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eaa1028ea14a0db7cb20235501ecd8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40033 } }
I20260806 09:13:58.986132 23052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:13:58.986308 23052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:13:58.986519 23052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:13:58.987274 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:13:58.963466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.3.69.62:40033)
0806 09:13:58.964266 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0806 09:13:58.964278 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:13:58.964320 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:13:58.966139 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:13:58.966161 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:13:58.966516 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:13:58.966823 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:13:58.968934 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:58.970335 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:58.972192 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:13:58.972838 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:13:58.972965 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:13:58.978089 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:13:58.978100 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:13:58.978118 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:13:58.978164 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:13:58.982405 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:58.983233 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:58.983247 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:58.983257 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:58.983482 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:13:58.983893 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:13:58.983898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:13:58.983901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:13:58.984659 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:13:58.984981 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 09:13:58.985504 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 09:13:58.985953 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":371,"threads_started":1}
I20260806 09:13:58.989720 23052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eaa1028ea14a0db7cb20235501ecd8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40033 } }
I20260806 09:13:58.990319 23052 leader_election.cc:302] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86eaa1028ea14a0db7cb20235501ecd8; no voters:
I20260806 09:13:58.990994 23052 leader_election.cc:288] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:13:58.991163 23058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:13:58.991837 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:13:58.993541 23052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:13:58.997867 23058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [term 1 LEADER]: Becoming Leader. State: Replica: 86eaa1028ea14a0db7cb20235501ecd8, State: Running, Role: LEADER
I20260806 09:13:58.998525 23058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eaa1028ea14a0db7cb20235501ecd8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40033 } }
I20260806 09:13:59.002014 23062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86eaa1028ea14a0db7cb20235501ecd8. Latest consensus state: current_term: 1 leader_uuid: "86eaa1028ea14a0db7cb20235501ecd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eaa1028ea14a0db7cb20235501ecd8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40033 } } }
I20260806 09:13:59.002112 23059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86eaa1028ea14a0db7cb20235501ecd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eaa1028ea14a0db7cb20235501ecd8" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40033 } } }
I20260806 09:13:59.002509 23062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [sys.catalog]: This master's current role is: LEADER
I20260806 09:13:59.002575 23059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8 [sys.catalog]: This master's current role is: LEADER
W20260806 09:13:59.010568 23079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:13:59.010771 23079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:13:59.010926 23080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:13:59.012771 23080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:13:59.021103 23080 catalog_manager.cc:1347] Generated new cluster ID: ce1fca627a944513970177a3accf46ca
I20260806 09:13:59.021224 23080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:13:59.032065 23080 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:13:59.033921 23080 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:13:59.042995 23080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86eaa1028ea14a0db7cb20235501ecd8: Generated new TSK 0
I20260806 09:13:59.043869 23080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:13:59.052860 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 22948
I20260806 09:13:59.054060 23044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:13:59.143756 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 22948
2026-08-06T09:13:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2255 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:13:59 dist-test-slave-w4v5 krb5kdc[23089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:13:59 dist-test-slave-w4v5 krb5kdc[23089](info): set up 2 sockets
Aug 06 09:13:59 dist-test-slave-w4v5 krb5kdc[23089](info): commencing operation
krb5kdc: starting...
W20260806 09:14:00.448853 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:14:00 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:14:00.734975 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37087
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:01.085661 23106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:01.086011 23106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:01.086269 23106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:01.093612 23106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:14:01.093720 23106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:01.093779 23106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:01.093830 23106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:14:01.093875 23106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:14:01.103540 23106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:37087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:01.104806 23106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:01.106077 23106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:01.118628 23112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:01.118633 23113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:01.118695 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:01.119536 23106 server_base.cc:1034] running on GCE node
I20260806 09:14:01.120301 23106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:01.121688 23106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:01.122908 23106 hybrid_clock.cc:648] HybridClock initialized: now 1786007641122850 us; error 45 us; skew 500 ppm
I20260806 09:14:01.123348 23106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:01 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007641, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:01.126573 23106 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:14:01.128494 23106 webserver.cc:460] Webserver started at http://127.3.69.62:42147/ using document root <none> and password file <none>
I20260806 09:14:01.129155 23106 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:01.129261 23106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:01.129519 23106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:14:01.132552 23106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "5110eeb691c44b3d838a41d30ea3d266"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "3edaac89e9b290bae08f9a4e46417d6d"
server_key_iv: "d4a849065276ee6b30acee7a7749d505"
server_key_version: "encryptionkey@0"
I20260806 09:14:01.133288 23106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "5110eeb691c44b3d838a41d30ea3d266"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "3edaac89e9b290bae08f9a4e46417d6d"
server_key_iv: "d4a849065276ee6b30acee7a7749d505"
server_key_version: "encryptionkey@0"
I20260806 09:14:01.138581 23106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 09:14:01.142074 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:01.143577 23106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:01.143769 23106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "5110eeb691c44b3d838a41d30ea3d266"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "3edaac89e9b290bae08f9a4e46417d6d"
server_key_iv: "d4a849065276ee6b30acee7a7749d505"
server_key_version: "encryptionkey@0"
I20260806 09:14:01.143976 23106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:01.162988 23106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:01.166893 23106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:01.167235 23106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:01.187654 23106 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:37087
I20260806 09:14:01.187706 23199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:37087 every 8 connection(s)
I20260806 09:14:01.189070 23106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:14:01.193068 23200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:01.194016 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 23106
I20260806 09:14:01.194294 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:14:01.195251 3348 external_mini_cluster.cc:1428] Setting key 14f086a3c398ba90caa5b0646c6b5747
I20260806 09:14:01.206207 23200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Bootstrap starting.
Aug 06 09:14:01 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:01.209671 23200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:01.210744 23200 log.cc:824] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:01.214036 23200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: No bootstrap required, opened a new log
I20260806 09:14:01.218327 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:01.197615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.3.69.62:37087)
0806 09:14:01.198410 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:01.198422 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:01.198462 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:14:01.200135 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:01.200157 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:01.200529 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:01.200944 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:01.202454 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:01.203846 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:01.205179 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:01.205836 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:01.206037 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:01.210569 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:01.210580 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:01.210594 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:01.210634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:01.214072 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:01.214771 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:01.214779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:01.214794 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:01.214963 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:01.215397 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:01.215402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:01.215407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:01.215976 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:01.216396 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:01.216872 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 09:14:01.217294 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":359,"threads_started":1}
I20260806 09:14:01.223038 23200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:01.223510 23200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:01.223629 23200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5110eeb691c44b3d838a41d30ea3d266, State: Initialized, Role: FOLLOWER
I20260806 09:14:01.224188 23200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:01.224442 23200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:14:01.224604 23200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:14:01.224807 23200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:01.227618 23200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:01.228150 23200 leader_election.cc:302] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5110eeb691c44b3d838a41d30ea3d266; no voters:
I20260806 09:14:01.228837 23200 leader_election.cc:288] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:14:01.228971 23209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:14:01.230209 23209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 1 LEADER]: Becoming Leader. State: Replica: 5110eeb691c44b3d838a41d30ea3d266, State: Running, Role: LEADER
I20260806 09:14:01.230808 23209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:01.231338 23200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:14:01.234081 23210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5110eeb691c44b3d838a41d30ea3d266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } } }
I20260806 09:14:01.234030 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5110eeb691c44b3d838a41d30ea3d266. Latest consensus state: current_term: 1 leader_uuid: "5110eeb691c44b3d838a41d30ea3d266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } } }
I20260806 09:14:01.234416 23210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:01.234512 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:01.236205 23217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:14:01.241626 23217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:14:01.250401 23217 catalog_manager.cc:1347] Generated new cluster ID: f38038106ddf441282b1b85c5ed53863
I20260806 09:14:01.250528 23217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:14:01.266958 23217 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:14:01.268923 23217 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:14:01.277688 23217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Generated new TSK 0
I20260806 09:14:01.278790 23217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:14:01.348989 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37087
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:01.698752 23238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:01.699040 23238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:01.699216 23238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:01.706550 23238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:01.706673 23238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:01.706904 23238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:14:01.717419 23238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:01.718933 23238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:01.720302 23238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:01.734042 23243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:01.734099 23246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:01.734077 23244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:01.740374 23238 server_base.cc:1034] running on GCE node
I20260806 09:14:01.741130 23238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:01.742172 23238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:01.743423 23238 hybrid_clock.cc:648] HybridClock initialized: now 1786007641743359 us; error 61 us; skew 500 ppm
I20260806 09:14:01.743814 23238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:01 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007641, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:01.747321 23238 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:14:01.749109 23238 webserver.cc:460] Webserver started at http://127.3.69.1:34221/ using document root <none> and password file <none>
I20260806 09:14:01.749850 23238 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:01.749959 23238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:01.750218 23238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:14:01.753378 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "b33700e8d1a842d8ba5c6de6c7c07548"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "a8804f1f593431f054557916c9c8cf1a"
server_key_iv: "cd5cd400567b6e2903a0e0cd4912683a"
server_key_version: "encryptionkey@0"
I20260806 09:14:01.754185 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "b33700e8d1a842d8ba5c6de6c7c07548"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "a8804f1f593431f054557916c9c8cf1a"
server_key_iv: "cd5cd400567b6e2903a0e0cd4912683a"
server_key_version: "encryptionkey@0"
I20260806 09:14:01.759940 23238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:14:01.763548 23254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:01.765017 23238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:14:01.765230 23238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "b33700e8d1a842d8ba5c6de6c7c07548"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "a8804f1f593431f054557916c9c8cf1a"
server_key_iv: "cd5cd400567b6e2903a0e0cd4912683a"
server_key_version: "encryptionkey@0"
I20260806 09:14:01.765460 23238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:01.795899 23238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:01.800664 23238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:01.800997 23238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:01.802078 23238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:01.803646 23238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:01.803731 23238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:01.803861 23238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:01.803933 23238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:01.840636 23238 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:38897
I20260806 09:14:01.840745 23425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:38897 every 8 connection(s)
I20260806 09:14:01.842010 23238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:14:01.850261 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 23238
I20260806 09:14:01.850551 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:14:01.851536 3348 external_mini_cluster.cc:1428] Setting key 82aa6535731e1bda7e7f533ce3e2e530
Aug 06 09:14:01 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007641, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:01.867538 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:01.844698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:53823 (local address 127.3.69.62:37087)
0806 09:14:01.845264 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:01.845270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:01.846187 (+ 917us) server_negotiation.cc:407] Connection header received
0806 09:14:01.848508 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:01.848515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:01.848625 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:01.848770 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:01.850982 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:01.851864 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:01.854146 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:01.854538 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:01.854659 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:01.859759 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:01.859771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:01.859780 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:01.859833 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:01.863716 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:01.864644 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:01.864652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:01.864659 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:01.864819 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:01.865312 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:01.865319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:01.865325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:01.865730 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:01.865888 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:01.866722 (+ 834us) server_negotiation.cc:299] Negotiation successful
0806 09:14:01.866995 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":305,"threads_started":1}
I20260806 09:14:01.868669 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:01.845117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:53823)
0806 09:14:01.845886 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:01.845930 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:01.848180 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:01.849061 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:01.849071 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:01.849743 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:01.850755 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:01.850789 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:01.852075 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:01.852081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:01.853951 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:01.853968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:01.854813 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:01.854825 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:01.855234 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:01.856140 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:01.856181 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:01.859488 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:01.863878 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:01.863890 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:01.863910 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:01.864509 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:01.864939 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:01.864946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:01.864952 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:01.865196 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:01.865861 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:01.865870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:01.866300 (+ 430us) client_negotiation.cc:769] Sending connection context
0806 09:14:01.866579 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 09:14:01.867158 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":331,"threads_started":1}
I20260806 09:14:01.870428 23427 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37087
I20260806 09:14:01.870795 23427 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:01.871784 23427 heartbeater.cc:511] Master 127.3.69.62:37087 requested a full tablet report, sending...
I20260806 09:14:01.874085 23145 ts_manager.cc:194] Registered new tserver with Master: b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897)
I20260806 09:14:01.875664 23145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:53823
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:14:01.922093 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37087
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:02.275870 23434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:02.276211 23434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:02.276398 23434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:02.283839 23434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:02.284018 23434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:02.284255 23434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:14:02.294381 23434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:02.295838 23434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:02.297235 23434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:02.311411 23440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:02.311676 23442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:02.312163 23434 server_base.cc:1034] running on GCE node
W20260806 09:14:02.312578 23444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:02.313261 23434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:02.314296 23434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:02.315528 23434 hybrid_clock.cc:648] HybridClock initialized: now 1786007642315469 us; error 56 us; skew 500 ppm
I20260806 09:14:02.315855 23434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:02 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:02.319320 23434 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:14:02.320833 23434 webserver.cc:460] Webserver started at http://127.3.69.2:42201/ using document root <none> and password file <none>
I20260806 09:14:02.321481 23434 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:02.321584 23434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:02.321909 23434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:14:02.324872 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "08ddc95cfea24968a64cc07858ecd9ff"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "df83fabbea96d081643ecb0ea1659600"
server_key_iv: "cc3daea686ed67803c6e7dd10264d060"
server_key_version: "encryptionkey@0"
I20260806 09:14:02.325563 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "08ddc95cfea24968a64cc07858ecd9ff"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "df83fabbea96d081643ecb0ea1659600"
server_key_iv: "cc3daea686ed67803c6e7dd10264d060"
server_key_version: "encryptionkey@0"
I20260806 09:14:02.330941 23434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:14:02.334818 23458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:02.336366 23434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:02.336549 23434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "08ddc95cfea24968a64cc07858ecd9ff"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "df83fabbea96d081643ecb0ea1659600"
server_key_iv: "cc3daea686ed67803c6e7dd10264d060"
server_key_version: "encryptionkey@0"
I20260806 09:14:02.336845 23434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:02.369036 23434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:02.373910 23434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:02.374179 23434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:02.375208 23434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:02.376576 23434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:02.376662 23434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:02.376750 23434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:02.376803 23434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:02.412214 23434 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:38499
I20260806 09:14:02.412302 23624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:38499 every 8 connection(s)
I20260806 09:14:02.413587 23434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:14:02.423952 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 23434
I20260806 09:14:02.424197 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:14:02.424774 3348 external_mini_cluster.cc:1428] Setting key f5a9d091c0bcfaab4e14e1248b4fbc2a
Aug 06 09:14:02 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:02.436408 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:02.416376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:55035 (local address 127.3.69.62:37087)
0806 09:14:02.416849 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:02.416854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:02.417911 (+ 1057us) server_negotiation.cc:407] Connection header received
0806 09:14:02.419923 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:02.419928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:02.420022 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:02.420160 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:02.421683 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:02.422416 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:02.424521 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:02.425023 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:02.425147 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:02.429211 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:02.429216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:02.429219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:02.429254 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:02.432233 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:02.433171 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:02.433180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:02.433186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:02.433369 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:02.433925 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:02.433930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:02.433934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:02.434382 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:02.434557 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:02.435594 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0806 09:14:02.435915 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
I20260806 09:14:02.437568 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:02.416935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:55035)
0806 09:14:02.417717 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:02.417761 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:02.419604 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:02.420400 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:02.420408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:02.420929 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:02.421491 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:02.421508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:02.422598 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:02.422603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:02.424361 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:02.424379 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:02.425273 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:02.425282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:02.425547 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:02.426272 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:02.426302 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:02.428966 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:02.432376 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:02.432386 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:02.432403 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:02.433015 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:02.433515 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:02.433520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:02.433523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:02.433791 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:02.434558 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:02.434570 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:02.435109 (+ 539us) client_negotiation.cc:769] Sending connection context
0806 09:14:02.435426 (+ 317us) client_negotiation.cc:240] Negotiation successful
0806 09:14:02.436045 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":337,"threads_started":1}
I20260806 09:14:02.440819 23625 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37087
I20260806 09:14:02.441215 23625 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:02.442116 23625 heartbeater.cc:511] Master 127.3.69.62:37087 requested a full tablet report, sending...
I20260806 09:14:02.444253 23145 ts_manager.cc:194] Registered new tserver with Master: 08ddc95cfea24968a64cc07858ecd9ff (127.3.69.2:38499)
I20260806 09:14:02.445583 23145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:55035
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:14:02.494935 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37087
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:02.843154 23633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:02.843550 23633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:02.843722 23633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:02.852576 23633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:02.852692 23633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:02.852811 23633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:14:02.862622 23633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:02.863933 23633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:02.865190 23633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:02.878613 23640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:02.878558 23643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:02.878649 23641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:02.879601 23427 heartbeater.cc:503] Master 127.3.69.62:37087 was elected leader, sending a full tablet report...
I20260806 09:14:02.879652 23633 server_base.cc:1034] running on GCE node
I20260806 09:14:02.880400 23633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:02.881706 23633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:02.882946 23633 hybrid_clock.cc:648] HybridClock initialized: now 1786007642882891 us; error 45 us; skew 500 ppm
I20260806 09:14:02.883301 23633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:02 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:02.886534 23633 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:14:02.888021 23633 webserver.cc:460] Webserver started at http://127.3.69.3:34037/ using document root <none> and password file <none>
I20260806 09:14:02.888695 23633 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:02.888845 23633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:02.889155 23633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:14:02.892129 23633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "1ca9ab304e28471e9a9efd767fb2da75"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "8c1e3a7f7ad1ab4245e9af7b2cd11d93"
server_key_iv: "91dc3b7acb61888901560925b4683654"
server_key_version: "encryptionkey@0"
I20260806 09:14:02.892891 23633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "1ca9ab304e28471e9a9efd767fb2da75"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "8c1e3a7f7ad1ab4245e9af7b2cd11d93"
server_key_iv: "91dc3b7acb61888901560925b4683654"
server_key_version: "encryptionkey@0"
I20260806 09:14:02.898072 23633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:14:02.901551 23656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:02.902874 23633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:14:02.903132 23633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "1ca9ab304e28471e9a9efd767fb2da75"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "8c1e3a7f7ad1ab4245e9af7b2cd11d93"
server_key_iv: "91dc3b7acb61888901560925b4683654"
server_key_version: "encryptionkey@0"
I20260806 09:14:02.903363 23633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:02.927616 23633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:02.931998 23633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:02.932320 23633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:02.933336 23633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:02.934736 23633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:02.934859 23633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:02.935009 23633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:02.935102 23633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:02.970667 23633 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:33247
I20260806 09:14:02.970757 23830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:33247 every 8 connection(s)
I20260806 09:14:02.972007 23633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:14:02.974524 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 23633
I20260806 09:14:02.974725 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:14:02.975435 3348 external_mini_cluster.cc:1428] Setting key a634105550fb81686fc3855106fb37b9
Aug 06 09:14:02 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:02.993009 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:02.974651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:39529 (local address 127.3.69.62:37087)
0806 09:14:02.975497 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:02.975503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:02.976589 (+ 1086us) server_negotiation.cc:407] Connection header received
0806 09:14:02.978636 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:02.978642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:02.978729 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:02.978832 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:02.980459 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:02.981217 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:02.982830 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:02.983290 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:02.983385 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:02.986896 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:02.986902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:02.986905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:02.986941 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:02.989730 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:02.990655 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:02.990661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:02.990664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:02.990863 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:02.991383 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:02.991387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:02.991389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:02.991711 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:02.991844 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:02.992507 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 09:14:02.992722 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":560,"threads_started":1}
I20260806 09:14:02.993827 23834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:02.975399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:39529)
0806 09:14:02.976405 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:02.976454 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:02.978342 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:02.979064 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:02.979125 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:02.979686 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:02.980303 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:02.980322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:02.981363 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:02.981367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:02.982701 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:02.982715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:02.983490 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:02.983495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:02.983721 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:02.984377 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:02.984400 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:02.986706 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:02.989870 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:02.989878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:02.989896 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:02.990525 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:02.990993 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:02.990997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:02.990999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:02.991277 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:02.991837 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:02.991842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:02.992141 (+ 299us) client_negotiation.cc:769] Sending connection context
0806 09:14:02.992377 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 09:14:02.992760 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":437,"threads_started":1}
I20260806 09:14:02.995225 23831 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37087
I20260806 09:14:02.995530 23831 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:02.996243 23831 heartbeater.cc:511] Master 127.3.69.62:37087 requested a full tablet report, sending...
I20260806 09:14:02.997653 23145 ts_manager.cc:194] Registered new tserver with Master: 1ca9ab304e28471e9a9efd767fb2da75 (127.3.69.3:33247)
I20260806 09:14:02.998493 23145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:39529
I20260806 09:14:03.003285 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:14:03.021901 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.008323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.3.69.62:37087)
0806 09:14:03.008616 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.008625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.009077 (+ 452us) server_negotiation.cc:407] Connection header received
0806 09:14:03.009295 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.009306 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.009384 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.009491 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:03.010924 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.011710 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.013210 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.013617 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.013756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.016313 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:03.016318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:03.016321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:03.016353 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:03.018991 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:03.019687 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:03.019692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:03.019696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:03.019883 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:03.020297 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:03.020302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:03.020306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:03.020637 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:03.020935 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.021332 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.021572 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260806 09:14:03.026295 23145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:14:03.033427 23145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:14:03.057669 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.047360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.3.69.2:38499)
0806 09:14:03.049569 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.049581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.049616 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:14:03.049740 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.049752 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.050067 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.050303 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:03.051297 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.052623 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.055530 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.056467 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.056634 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.056857 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.057170 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.057346 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2001,"thread_start_us":1874,"threads_started":1}
I20260806 09:14:03.057765 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.046856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:38499 (local address 127.0.0.1:45292)
0806 09:14:03.047975 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:03.048002 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:03.048202 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:03.050349 (+ 2147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:03.050360 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:03.050419 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:03.050994 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.051011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.052771 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.052788 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.055372 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.055384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.056721 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.056728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.056927 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.056965 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:14:03.057098 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 09:14:03.057296 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":4,"thread_start_us":805,"threads_started":1}
I20260806 09:14:03.059321 23851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.046154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:41060)
0806 09:14:03.046931 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:03.046996 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:03.047341 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:03.051749 (+ 4408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:03.051756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:03.051777 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:03.052188 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.052202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.054207 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.054211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.056541 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.056556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.058531 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.058540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.058728 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.058760 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:14:03.058832 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 09:14:03.058919 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":3,"thread_start_us":308,"threads_started":1}
I20260806 09:14:03.060530 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.046546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.3.69.1:38897)
0806 09:14:03.051149 (+ 4603us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.051157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.051188 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:14:03.051291 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.051298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.051534 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.052396 (+ 862us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:03.052643 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.053983 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.056736 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.058280 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.059684 (+ 1404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.059920 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.059986 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.060168 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4387,"thread_start_us":4233,"threads_started":1}
I20260806 09:14:03.061023 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.047306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:33247 (local address 127.0.0.1:36598)
0806 09:14:03.051439 (+ 4133us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:03.051462 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:03.051698 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:03.052687 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:03.052695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:03.052724 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:03.053305 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.053321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.055315 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.055320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.057579 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.057597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.060247 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.060251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.060372 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.060392 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 09:14:03.060496 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 09:14:03.060636 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4019,"mutex_wait_us":84,"thread_start_us":3881,"threads_started":1}
I20260806 09:14:03.062182 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.047941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.3.69.3:33247)
0806 09:14:03.049058 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.049072 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.051577 (+ 2505us) server_negotiation.cc:407] Connection header received
0806 09:14:03.051901 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.051910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.052146 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.052316 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:03.053513 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.055147 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.058571 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.059858 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.059944 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.060093 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.060543 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.060878 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":593,"threads_started":1}
I20260806 09:14:03.062341 23534 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb1227bfceb45f7a4139e3d96c98969 (DEFAULT_TABLE table=test-table [id=89ee2b9e78d14262b853fcf1868e9850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:03.063800 23534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb1227bfceb45f7a4139e3d96c98969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:03.064905 23325 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb1227bfceb45f7a4139e3d96c98969 (DEFAULT_TABLE table=test-table [id=89ee2b9e78d14262b853fcf1868e9850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:03.065522 23726 tablet_service.cc:1467] Processing CreateTablet for tablet 0bb1227bfceb45f7a4139e3d96c98969 (DEFAULT_TABLE table=test-table [id=89ee2b9e78d14262b853fcf1868e9850]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:03.066349 23325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb1227bfceb45f7a4139e3d96c98969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:03.066875 23726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bb1227bfceb45f7a4139e3d96c98969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:03.076673 23861 tablet_bootstrap.cc:492] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Bootstrap starting.
I20260806 09:14:03.079691 23861 tablet_bootstrap.cc:654] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:03.080942 23861 log.cc:824] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:03.083563 23862 tablet_bootstrap.cc:492] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Bootstrap starting.
I20260806 09:14:03.084000 23864 tablet_bootstrap.cc:492] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Bootstrap starting.
I20260806 09:14:03.085212 23861 tablet_bootstrap.cc:492] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: No bootstrap required, opened a new log
I20260806 09:14:03.085518 23861 ts_tablet_manager.cc:1397] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 09:14:03.086867 23864 tablet_bootstrap.cc:654] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:03.087218 23862 tablet_bootstrap.cc:654] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:03.088006 23864 log.cc:824] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:03.088418 23862 log.cc:824] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:03.091471 23864 tablet_bootstrap.cc:492] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: No bootstrap required, opened a new log
I20260806 09:14:03.091473 23862 tablet_bootstrap.cc:492] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: No bootstrap required, opened a new log
I20260806 09:14:03.091768 23864 ts_tablet_manager.cc:1397] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 09:14:03.091768 23862 ts_tablet_manager.cc:1397] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 09:14:03.094587 23861 raft_consensus.cc:348] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.094980 23861 raft_consensus.cc:374] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:03.095105 23861 raft_consensus.cc:729] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ddc95cfea24968a64cc07858ecd9ff, State: Initialized, Role: FOLLOWER
I20260806 09:14:03.095726 23861 consensus_queue.cc:260] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.096889 23625 heartbeater.cc:503] Master 127.3.69.62:37087 was elected leader, sending a full tablet report...
I20260806 09:14:03.097596 23861 ts_tablet_manager.cc:1428] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Time spent starting tablet: real 0.012s user 0.005s sys 0.008s
I20260806 09:14:03.102751 23862 raft_consensus.cc:348] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.102751 23864 raft_consensus.cc:348] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.103240 23864 raft_consensus.cc:374] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:03.103235 23862 raft_consensus.cc:374] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:03.103358 23864 raft_consensus.cc:729] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca9ab304e28471e9a9efd767fb2da75, State: Initialized, Role: FOLLOWER
I20260806 09:14:03.103350 23862 raft_consensus.cc:729] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33700e8d1a842d8ba5c6de6c7c07548, State: Initialized, Role: FOLLOWER
I20260806 09:14:03.103978 23862 consensus_queue.cc:260] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.103979 23864 consensus_queue.cc:260] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.105786 23831 heartbeater.cc:503] Master 127.3.69.62:37087 was elected leader, sending a full tablet report...
I20260806 09:14:03.106123 23862 ts_tablet_manager.cc:1428] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 09:14:03.106972 23864 ts_tablet_manager.cc:1428] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260806 09:14:03.165746 23626 tablet.cc:2369] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:14:03.223706 23832 tablet.cc:2369] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:14:03.344830 23428 tablet.cc:2369] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:14:03.391297 23867 raft_consensus.cc:482] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:14:03.391642 23867 raft_consensus.cc:504] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.393388 23867 leader_election.cc:288] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897), 1ca9ab304e28471e9a9efd767fb2da75 (127.3.69.3:33247)
I20260806 09:14:03.398697 23869 raft_consensus.cc:482] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:14:03.399096 23869 raft_consensus.cc:504] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.401499 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.393926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:32941 (local address 127.3.69.1:38897)
0806 09:14:03.394137 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.394142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.394795 (+ 653us) server_negotiation.cc:407] Connection header received
0806 09:14:03.395095 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.395104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.395203 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.395300 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:03.396100 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.397071 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.399346 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.400351 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.400503 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.400616 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.401090 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.401237 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:14:03.401522 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.393911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.3.69.2:32941)
0806 09:14:03.394671 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:03.394694 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:03.394916 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:03.395373 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:03.395378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:03.395416 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:03.395878 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.395891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.397212 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.397218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.399177 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.399191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.400602 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.400609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.400837 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.400876 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:14:03.400977 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 09:14:03.401116 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":400,"threads_started":1}
I20260806 09:14:03.402303 23869 leader_election.cc:288] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897), 08ddc95cfea24968a64cc07858ecd9ff (127.3.69.2:38499)
I20260806 09:14:03.402757 23354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1227bfceb45f7a4139e3d96c98969" candidate_uuid: "08ddc95cfea24968a64cc07858ecd9ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" is_pre_election: true
I20260806 09:14:03.403457 23354 raft_consensus.cc:2454] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ddc95cfea24968a64cc07858ecd9ff in term 0.
I20260806 09:14:03.404018 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.394951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:49499 (local address 127.3.69.3:33247)
0806 09:14:03.395230 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.395239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.395703 (+ 464us) server_negotiation.cc:407] Connection header received
0806 09:14:03.395798 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.395805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.395889 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.396008 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:03.396692 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.397705 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.400122 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.401449 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.401607 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.401716 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.403633 (+ 1917us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.403767 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 09:14:03.404333 23461 leader_election.cc:302] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ddc95cfea24968a64cc07858ecd9ff, b33700e8d1a842d8ba5c6de6c7c07548; no voters:
I20260806 09:14:03.404680 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.394696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:33247 (local address 127.3.69.2:49499)
0806 09:14:03.395583 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:03.395597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:03.395699 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:03.396080 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:03.396085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:03.396125 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:03.396484 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.396495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.397883 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.397887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.399928 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.399943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.402152 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.402158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.403407 (+ 1249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.403445 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:14:03.404180 (+ 735us) client_negotiation.cc:240] Negotiation successful
0806 09:14:03.404339 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":538,"threads_started":1}
I20260806 09:14:03.404865 23867 raft_consensus.cc:2781] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:14:03.405064 23867 raft_consensus.cc:482] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:14:03.405216 23867 raft_consensus.cc:3037] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:03.408082 23867 raft_consensus.cc:504] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.408756 23867 leader_election.cc:288] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [CANDIDATE]: Term 1 election: Requested vote from peers b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897), 1ca9ab304e28471e9a9efd767fb2da75 (127.3.69.3:33247)
I20260806 09:14:03.409433 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.402094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:50999 (local address 127.3.69.1:38897)
0806 09:14:03.402286 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.402294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.402313 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:14:03.402378 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.402382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.402466 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.403435 (+ 969us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:03.403594 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.404732 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.407206 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.408448 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.408583 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.408732 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.409036 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.409162 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:14:03.409433 23834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.401537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.3.69.3:50999)
0806 09:14:03.401844 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:03.401864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:03.401984 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:03.402871 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:03.402877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:03.402910 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:03.403273 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.403287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.404892 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.404899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.406977 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.406991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.408627 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.408633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.408857 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.408899 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 09:14:03.409005 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 09:14:03.409142 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260806 09:14:03.410071 23757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1227bfceb45f7a4139e3d96c98969" candidate_uuid: "08ddc95cfea24968a64cc07858ecd9ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca9ab304e28471e9a9efd767fb2da75" is_pre_election: true
I20260806 09:14:03.410171 23755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1227bfceb45f7a4139e3d96c98969" candidate_uuid: "08ddc95cfea24968a64cc07858ecd9ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca9ab304e28471e9a9efd767fb2da75"
I20260806 09:14:03.410336 23354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1227bfceb45f7a4139e3d96c98969" candidate_uuid: "1ca9ab304e28471e9a9efd767fb2da75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" is_pre_election: true
I20260806 09:14:03.410580 23354 raft_consensus.cc:2454] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca9ab304e28471e9a9efd767fb2da75 in term 0.
I20260806 09:14:03.410626 23755 raft_consensus.cc:3037] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:03.411249 23665 leader_election.cc:302] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca9ab304e28471e9a9efd767fb2da75, b33700e8d1a842d8ba5c6de6c7c07548; no voters:
I20260806 09:14:03.411520 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.402664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:59603 (local address 127.3.69.2:38499)
0806 09:14:03.402872 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.402878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.403533 (+ 655us) server_negotiation.cc:407] Connection header received
0806 09:14:03.403910 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.403916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.403988 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.404053 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:03.405284 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.406418 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.408892 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.410177 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.410332 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.410433 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.411054 (+ 621us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.411223 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:14:03.412307 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.402588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:38499 (local address 127.3.69.3:59603)
0806 09:14:03.403449 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:03.403466 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:03.403629 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:03.404384 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:03.404389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:03.404414 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:03.405050 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.405064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.406619 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.406624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.408694 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:03.408721 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.410615 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.410622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:03.410827 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.410870 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:14:03.411765 (+ 895us) client_negotiation.cc:240] Negotiation successful
0806 09:14:03.411912 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":429,"threads_started":1}
I20260806 09:14:03.412863 23354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1227bfceb45f7a4139e3d96c98969" candidate_uuid: "08ddc95cfea24968a64cc07858ecd9ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33700e8d1a842d8ba5c6de6c7c07548"
I20260806 09:14:03.413093 23354 raft_consensus.cc:3037] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:03.413478 23562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bb1227bfceb45f7a4139e3d96c98969" candidate_uuid: "1ca9ab304e28471e9a9efd767fb2da75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ddc95cfea24968a64cc07858ecd9ff" is_pre_election: true
I20260806 09:14:03.414036 23562 raft_consensus.cc:2379] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ca9ab304e28471e9a9efd767fb2da75 in current term 1: Already voted for candidate 08ddc95cfea24968a64cc07858ecd9ff in this term.
I20260806 09:14:03.414402 23755 raft_consensus.cc:2454] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ddc95cfea24968a64cc07858ecd9ff in term 1.
I20260806 09:14:03.414553 23869 raft_consensus.cc:2741] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260806 09:14:03.415221 23461 leader_election.cc:302] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ddc95cfea24968a64cc07858ecd9ff, 1ca9ab304e28471e9a9efd767fb2da75; no voters:
I20260806 09:14:03.416301 23867 raft_consensus.cc:2781] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:14:03.416575 23354 raft_consensus.cc:2454] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ddc95cfea24968a64cc07858ecd9ff in term 1.
I20260806 09:14:03.418114 23867 raft_consensus.cc:686] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 LEADER]: Becoming Leader. State: Replica: 08ddc95cfea24968a64cc07858ecd9ff, State: Running, Role: LEADER
I20260806 09:14:03.418666 23867 consensus_queue.cc:237] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } }
I20260806 09:14:03.423635 23145 catalog_manager.cc:5495] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 08ddc95cfea24968a64cc07858ecd9ff (127.3.69.2). New cstate: current_term: 1 leader_uuid: "08ddc95cfea24968a64cc07858ecd9ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } health_report { overall_health: HEALTHY } } }
I20260806 09:14:03.519737 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.513674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.3.69.2:38499)
0806 09:14:03.513888 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.513892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.513984 (+ 92us) server_negotiation.cc:407] Connection header received
0806 09:14:03.514249 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.514253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.514363 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.514489 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:14:03.514919 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.515793 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.517251 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.517820 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.517928 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.518485 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:14:03.519026 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:14:03.519260 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.519313 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.519387 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:14:03.528530 23755 raft_consensus.cc:1261] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 FOLLOWER]: Refusing update from remote peer 08ddc95cfea24968a64cc07858ecd9ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:14:03.528538 23354 raft_consensus.cc:1261] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 1 FOLLOWER]: Refusing update from remote peer 08ddc95cfea24968a64cc07858ecd9ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:14:03.529450 23874 consensus_queue.cc:1035] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:14:03.529836 23867 consensus_queue.cc:1035] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:14:03.542397 23882 mvcc.cc:205] Tried to move back new op lower bound from 7315487307881590784 to 7315487307447410688. Current Snapshot: MvccSnapshot[applied={T|T < 7315487307881590784}]
I20260806 09:14:03.557091 23144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:14:03.557427 23144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:14:03.561137 23144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Sending DeleteTablet for 3 replicas of tablet 0bb1227bfceb45f7a4139e3d96c98969
I20260806 09:14:03.562418 23325 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb1227bfceb45f7a4139e3d96c98969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:03 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:41060
I20260806 09:14:03.562690 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 23238
I20260806 09:14:03.562603 23726 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb1227bfceb45f7a4139e3d96c98969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:03 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:36598
I20260806 09:14:03.563655 23534 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bb1227bfceb45f7a4139e3d96c98969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:03 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:45292
I20260806 09:14:03.573499 23898 tablet_replica.cc:330] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: stopping tablet replica
I20260806 09:14:03.574208 23898 raft_consensus.cc:2229] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:14:03.574863 23898 raft_consensus.cc:2258] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:03.576176 23900 tablet_replica.cc:330] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: stopping tablet replica
I20260806 09:14:03.576885 23900 raft_consensus.cc:2229] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:14:03.577505 23898 ts_tablet_manager.cc:1905] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:03.577841 23900 raft_consensus.cc:2258] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:03.580384 23900 ts_tablet_manager.cc:1905] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:03.585453 23898 ts_tablet_manager.cc:1918] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:03.585750 23898 log.cc:1196] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/0bb1227bfceb45f7a4139e3d96c98969
I20260806 09:14:03.586480 23898 ts_tablet_manager.cc:1939] T 0bb1227bfceb45f7a4139e3d96c98969 P b33700e8d1a842d8ba5c6de6c7c07548: Deleting consensus metadata
I20260806 09:14:03.588757 23129 catalog_manager.cc:4841] TS b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897): tablet 0bb1227bfceb45f7a4139e3d96c98969 (table test-table [id=89ee2b9e78d14262b853fcf1868e9850]) successfully deleted
I20260806 09:14:03.590225 23901 tablet_replica.cc:330] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: stopping tablet replica
I20260806 09:14:03.590252 23900 ts_tablet_manager.cc:1918] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:03.590582 23900 log.cc:1196] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/0bb1227bfceb45f7a4139e3d96c98969
I20260806 09:14:03.590976 23901 raft_consensus.cc:2229] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:14:03.591619 23901 raft_consensus.cc:2258] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:03.591652 23900 ts_tablet_manager.cc:1939] T 0bb1227bfceb45f7a4139e3d96c98969 P 08ddc95cfea24968a64cc07858ecd9ff: Deleting consensus metadata
I20260806 09:14:03.593745 23127 catalog_manager.cc:4841] TS 08ddc95cfea24968a64cc07858ecd9ff (127.3.69.2:38499): tablet 0bb1227bfceb45f7a4139e3d96c98969 (table test-table [id=89ee2b9e78d14262b853fcf1868e9850]) successfully deleted
I20260806 09:14:03.593909 23901 ts_tablet_manager.cc:1905] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 09:14:03 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:14:03.601117 23901 ts_tablet_manager.cc:1918] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:03.601290 23901 log.cc:1196] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/0bb1227bfceb45f7a4139e3d96c98969
I20260806 09:14:03.601863 23901 ts_tablet_manager.cc:1939] T 0bb1227bfceb45f7a4139e3d96c98969 P 1ca9ab304e28471e9a9efd767fb2da75: Deleting consensus metadata
I20260806 09:14:03.603858 23129 catalog_manager.cc:4841] TS 1ca9ab304e28471e9a9efd767fb2da75 (127.3.69.3:33247): tablet 0bb1227bfceb45f7a4139e3d96c98969 (table test-table [id=89ee2b9e78d14262b853fcf1868e9850]) successfully deleted
I20260806 09:14:03.607363 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.564344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.3.69.1:38897)
0806 09:14:03.564585 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.564593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.589472 (+ 24879us) server_negotiation.cc:407] Connection header received
0806 09:14:03.589710 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.589714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.589805 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.589917 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:03.592305 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.593372 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.595151 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.595623 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.595791 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.600329 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:03.600343 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:03.600358 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:03.600402 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:03.603642 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:03.604481 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:03.604491 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:03.604495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:03.604676 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:03.605266 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:03.605270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:03.605272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:03.605787 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:03.606113 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.606632 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.606960 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:14:03.608155 23419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:03.700033 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 23238
I20260806 09:14:03.730994 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 23434
Aug 06 09:14:03 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:14:03.749496 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.732007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.3.69.2:38499)
0806 09:14:03.732274 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.732281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.732303 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:14:03.732382 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.732388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.732496 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.732605 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:03.733890 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.735139 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.736626 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.737333 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.737461 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.741908 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:03.741932 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:03.741959 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:03.742015 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:03.746034 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:03.746764 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:03.746773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:03.746777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:03.747000 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:03.747679 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:03.747684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:03.747688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:03.748192 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:03.748525 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.748840 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.749096 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260806 09:14:03.750326 23618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:03.869702 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 23434
I20260806 09:14:03.899757 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 23633
Aug 06 09:14:03 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:14:03.920842 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:03.900626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.3.69.3:33247)
0806 09:14:03.900804 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:03.900809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:03.900827 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:14:03.900953 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:03.900957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:03.901035 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:03.901128 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:03.902559 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.903479 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.905141 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:03.905564 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:03.905676 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:03.914617 (+ 8941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:03.914638 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:03.914656 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:03.914703 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:03.917857 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:03.918655 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:03.918663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:03.918665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:03.918824 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:03.919307 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:03.919311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:03.919313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:03.919721 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:03.919957 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:03.920309 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 09:14:03.920495 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:14:03.921563 23823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:04.014726 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 23633
I20260806 09:14:04.041994 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 23106
I20260806 09:14:04.043335 23195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:04.156246 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 23106
I20260806 09:14:04.188519 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37087
--webserver_interface=127.3.69.62
--webserver_port=42147
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:04.540756 23910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:04.541050 23910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:04.541236 23910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:04.549255 23910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:14:04.549367 23910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:04.549486 23910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:04.549587 23910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:14:04.549656 23910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:14:04.559589 23910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:37087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=42147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:04.560961 23910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:04.562340 23910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:04.591801 23918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:04.591907 23927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:04.591835 23919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:04.592737 23910 server_base.cc:1034] running on GCE node
I20260806 09:14:04.593758 23910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:04.595477 23910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:04.596742 23910 hybrid_clock.cc:648] HybridClock initialized: now 1786007644596677 us; error 74 us; skew 500 ppm
I20260806 09:14:04.597169 23910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:04 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007644, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:04.601007 23910 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:14:04.602820 23910 webserver.cc:460] Webserver started at http://127.3.69.62:42147/ using document root <none> and password file <none>
I20260806 09:14:04.603658 23910 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:04.603832 23910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:04.610438 23910 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 09:14:04.613618 23939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:04.614941 23910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:14:04.615186 23910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "5110eeb691c44b3d838a41d30ea3d266"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "3edaac89e9b290bae08f9a4e46417d6d"
server_key_iv: "d4a849065276ee6b30acee7a7749d505"
server_key_version: "encryptionkey@0"
I20260806 09:14:04.616276 23910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:04.652817 23910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:04.657375 23910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:04.657697 23910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:04.678362 23910 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:37087
I20260806 09:14:04.678449 24017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:37087 every 8 connection(s)
I20260806 09:14:04.680070 23910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:14:04.687325 24018 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:14:04.689839 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 23910
I20260806 09:14:04.690954 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:38897
--local_ip_for_outbound_sockets=127.3.69.1
--tserver_master_addrs=127.3.69.62:37087
--webserver_port=34221
--webserver_interface=127.3.69.1
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:14:04.691560 24018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Bootstrap starting.
I20260806 09:14:04.716321 24018 log.cc:824] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:04.729656 24018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:14:04.730270 24018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Bootstrap complete.
I20260806 09:14:04.740097 24018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:04.741027 24018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5110eeb691c44b3d838a41d30ea3d266, State: Initialized, Role: FOLLOWER
I20260806 09:14:04.741667 24018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:04.741950 24018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:14:04.742106 24018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:14:04.742300 24018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:14:04.746409 24018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:04.746939 24018 leader_election.cc:302] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5110eeb691c44b3d838a41d30ea3d266; no voters:
I20260806 09:14:04.747721 24018 leader_election.cc:288] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:14:04.747872 24027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:14:04.749253 24027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 2 LEADER]: Becoming Leader. State: Replica: 5110eeb691c44b3d838a41d30ea3d266, State: Running, Role: LEADER
I20260806 09:14:04.749877 24027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:04.750240 24018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:14:04.752708 24028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5110eeb691c44b3d838a41d30ea3d266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } } }
I20260806 09:14:04.753149 24028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:04.754966 24029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5110eeb691c44b3d838a41d30ea3d266. Latest consensus state: current_term: 2 leader_uuid: "5110eeb691c44b3d838a41d30ea3d266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } } }
I20260806 09:14:04.755363 24029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:04.755589 24032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:14:04.780732 24032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:14:04.782599 24032 catalog_manager.cc:1259] Loaded cluster ID: f38038106ddf441282b1b85c5ed53863
I20260806 09:14:04.782735 24032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:14:04.787631 24032 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 09:14:04.811764 24032 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 09:14:04.835132 24032 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:14:04.836998 24032 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Loaded TSK: 0
I20260806 09:14:04.838578 24032 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:14:05.082046 24020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:05.082366 24020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:05.082531 24020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:05.089962 24020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:05.090134 24020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:05.090313 24020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:14:05.100523 24020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:38897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=34221
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:05.101879 24020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:05.103345 24020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:05.118172 24059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:05.118515 24060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:05.119278 24063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:05.119553 24020 server_base.cc:1034] running on GCE node
I20260806 09:14:05.120258 24020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:05.121342 24020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:05.122579 24020 hybrid_clock.cc:648] HybridClock initialized: now 1786007645122512 us; error 57 us; skew 500 ppm
I20260806 09:14:05.122925 24020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:05 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007645, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:05.126407 24020 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:14:05.127979 24020 webserver.cc:460] Webserver started at http://127.3.69.1:34221/ using document root <none> and password file <none>
I20260806 09:14:05.128720 24020 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:05.128872 24020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:05.134534 24020 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:14:05.137547 24076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:05.139027 24020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:14:05.139276 24020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "b33700e8d1a842d8ba5c6de6c7c07548"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "a8804f1f593431f054557916c9c8cf1a"
server_key_iv: "cd5cd400567b6e2903a0e0cd4912683a"
server_key_version: "encryptionkey@0"
I20260806 09:14:05.140300 24020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:05.159260 24020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:05.163923 24020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:05.164302 24020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:05.165428 24020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:05.166797 24020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:05.166944 24020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:05.167119 24020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:05.167212 24020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 09:14:05.204470 24020 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:38897
I20260806 09:14:05.204566 24237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:38897 every 8 connection(s)
I20260806 09:14:05.206070 24020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:14:05.215758 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 24020
I20260806 09:14:05.216544 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:38499
--local_ip_for_outbound_sockets=127.3.69.2
--tserver_master_addrs=127.3.69.62:37087
--webserver_port=42201
--webserver_interface=127.3.69.2
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:14:05 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007645, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:05.233307 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:05.209093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:44849 (local address 127.3.69.62:37087)
0806 09:14:05.209717 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:05.209728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:05.210393 (+ 665us) server_negotiation.cc:407] Connection header received
0806 09:14:05.212675 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:05.212689 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:05.213015 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:05.213317 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:05.214836 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:05.217004 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:05.218738 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:05.219841 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:05.220044 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:05.223891 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:05.223904 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:05.223928 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:05.223980 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:05.228228 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:05.229315 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:05.229322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:05.229327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:05.229494 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:05.230189 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:05.230193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:05.230195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:05.230738 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:05.230900 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:05.231966 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0806 09:14:05.232308 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":282,"threads_started":1}
I20260806 09:14:05.233701 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:05.209285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:44849)
0806 09:14:05.210201 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:05.210255 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:05.212126 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:05.213494 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:05.213502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:05.214029 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:05.214589 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:05.214605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:05.217180 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:05.217185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:05.218580 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:05.218593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:05.220108 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:05.220114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:05.220377 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:05.221157 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:05.221180 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:05.223587 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:05.228441 (+ 4854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:05.228454 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:05.228473 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:05.229167 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:05.229667 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:05.229673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:05.229677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:05.229936 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:05.230904 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:05.230915 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:05.231397 (+ 482us) client_negotiation.cc:769] Sending connection context
0806 09:14:05.231725 (+ 328us) client_negotiation.cc:240] Negotiation successful
0806 09:14:05.232338 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":386,"threads_started":1}
I20260806 09:14:05.236737 24238 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37087
I20260806 09:14:05.237066 24238 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:05.237975 24238 heartbeater.cc:511] Master 127.3.69.62:37087 requested a full tablet report, sending...
I20260806 09:14:05.240455 23960 ts_manager.cc:194] Registered new tserver with Master: b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897)
I20260806 09:14:05.241955 23960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:44849
W20260806 09:14:05.574481 24244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:05.574819 24244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:05.574985 24244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:05.585649 24244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:05.585840 24244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:05.586141 24244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:14:05.601048 24244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:38499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=42201
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:05.602319 24244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:05.603720 24244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:05.617617 24256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:05.617547 24258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:05.617654 24254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:05.618508 24244 server_base.cc:1034] running on GCE node
I20260806 09:14:05.619241 24244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:05.620198 24244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:05.621443 24244 hybrid_clock.cc:648] HybridClock initialized: now 1786007645621402 us; error 35 us; skew 500 ppm
I20260806 09:14:05.621762 24244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:05 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007645, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:05.624961 24244 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:14:05.626516 24244 webserver.cc:460] Webserver started at http://127.3.69.2:42201/ using document root <none> and password file <none>
I20260806 09:14:05.627313 24244 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:05.627483 24244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:05.632962 24244 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:14:05.635655 24268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:05.636968 24244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:05.637162 24244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "08ddc95cfea24968a64cc07858ecd9ff"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "df83fabbea96d081643ecb0ea1659600"
server_key_iv: "cc3daea686ed67803c6e7dd10264d060"
server_key_version: "encryptionkey@0"
I20260806 09:14:05.638152 24244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:05.652977 24244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:05.656847 24244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:05.657123 24244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:05.658053 24244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:05.659458 24244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:05.659586 24244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:05.659722 24244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:05.659813 24244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:05.694886 24244 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:38499
I20260806 09:14:05.694950 24445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:38499 every 8 connection(s)
I20260806 09:14:05.696619 24244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:14:05.701663 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 24244
I20260806 09:14:05.702356 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:33247
--local_ip_for_outbound_sockets=127.3.69.3
--tserver_master_addrs=127.3.69.62:37087
--webserver_port=34037
--webserver_interface=127.3.69.3
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:14:05 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007645, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:05.721441 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:05.699586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:35289 (local address 127.3.69.62:37087)
0806 09:14:05.699807 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:05.699814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:05.700994 (+ 1180us) server_negotiation.cc:407] Connection header received
0806 09:14:05.703013 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:05.703019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:05.703163 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:05.703282 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:05.704826 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:05.705772 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:05.708332 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:05.709024 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:05.709152 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:05.713993 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:05.714001 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:05.714008 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:05.714055 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:05.717828 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:05.718574 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:05.718583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:05.718588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:05.718793 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:05.719277 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:05.719284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:05.719290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:05.719695 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:05.719888 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:05.720688 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 09:14:05.720963 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:14:05.722218 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:05.700074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:35289)
0806 09:14:05.700819 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:05.700865 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:05.702729 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:05.703543 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:05.703552 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:05.704065 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:05.704648 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:05.704664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:05.705955 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:05.705963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:05.708174 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:05.708195 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:05.709275 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:05.709284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:05.709627 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:05.710449 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:05.710502 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:05.713774 (+ 3272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:05.717995 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:05.718002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:05.718015 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:05.718441 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:05.718910 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:05.718914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:05.718916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:05.719163 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:05.719867 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:05.719872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:05.720214 (+ 342us) client_negotiation.cc:769] Sending connection context
0806 09:14:05.720549 (+ 335us) client_negotiation.cc:240] Negotiation successful
0806 09:14:05.721026 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":299,"threads_started":1}
I20260806 09:14:05.723922 24448 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37087
I20260806 09:14:05.724263 24448 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:05.725028 24448 heartbeater.cc:511] Master 127.3.69.62:37087 requested a full tablet report, sending...
I20260806 09:14:05.726555 23960 ts_manager.cc:194] Registered new tserver with Master: 08ddc95cfea24968a64cc07858ecd9ff (127.3.69.2:38499)
I20260806 09:14:05.727432 23960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:35289
W20260806 09:14:06.051097 24451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:06.051435 24451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:06.051673 24451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:06.059013 24451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:06.059203 24451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:06.059391 24451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:14:06.069204 24451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:33247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=34037
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:06.070585 24451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:06.071949 24451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:06.085139 24462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:06.085254 24465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:06.085494 24458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:06.086440 24451 server_base.cc:1034] running on GCE node
I20260806 09:14:06.087146 24451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:06.088049 24451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:06.089298 24451 hybrid_clock.cc:648] HybridClock initialized: now 1786007646089261 us; error 46 us; skew 500 ppm
I20260806 09:14:06.089627 24451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:06 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007646, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:06.092888 24451 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:14:06.094353 24451 webserver.cc:460] Webserver started at http://127.3.69.3:34037/ using document root <none> and password file <none>
I20260806 09:14:06.095041 24451 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:06.095263 24451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:06.100863 24451 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:14:06.103726 24472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:06.105121 24451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:06.105383 24451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "1ca9ab304e28471e9a9efd767fb2da75"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "8c1e3a7f7ad1ab4245e9af7b2cd11d93"
server_key_iv: "91dc3b7acb61888901560925b4683654"
server_key_version: "encryptionkey@0"
I20260806 09:14:06.106402 24451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:06.118896 24451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:06.123203 24451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:06.123490 24451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:06.124496 24451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:06.125938 24451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:06.126065 24451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:06.126209 24451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:06.126318 24451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:06.162374 24451 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:33247
I20260806 09:14:06.162472 24636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:33247 every 8 connection(s)
I20260806 09:14:06.164007 24451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:14:06.166785 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 24451
Aug 06 09:14:06 dist-test-slave-w4v5 krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007646, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:06.186311 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.167831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.3.69.62:37087)
0806 09:14:06.171503 (+ 3672us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.171512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.171888 (+ 376us) server_negotiation.cc:407] Connection header received
0806 09:14:06.172097 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.172106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.172165 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.172275 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:06.173841 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.174634 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.176102 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.176671 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.176803 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.179745 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:06.179750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:06.179753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:06.179783 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:06.183042 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.183959 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.183968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.183974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.184114 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.184544 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.184547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.184549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.184976 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:06.185190 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.185591 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.185839 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":3547,"spinlock_wait_cycles":896,"thread_start_us":3456,"threads_started":1}
I20260806 09:14:06.186975 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.166633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:59153 (local address 127.3.69.62:37087)
0806 09:14:06.166824 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.166828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.168533 (+ 1705us) server_negotiation.cc:407] Connection header received
0806 09:14:06.172003 (+ 3470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.172009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.172094 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.172220 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:06.173686 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.174453 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.176049 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.176569 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.176720 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.180466 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:06.180471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:06.180474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:06.180510 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:06.183497 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.184366 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.184371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.184374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.184528 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.185001 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.185005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.185006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.185379 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:06.185515 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.186340 (+ 825us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.186573 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":20608}
I20260806 09:14:06.188139 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.167321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:59153)
0806 09:14:06.168333 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.168393 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:06.171709 (+ 3316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:06.172456 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:06.172464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:06.172949 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:06.173503 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.173520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.174614 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.174619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.175897 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.175909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.176812 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.176820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.177068 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.177840 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:06.177861 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:06.180252 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:06.183641 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:06.183647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:06.183658 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:06.184230 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:06.184648 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:06.184655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:06.184661 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:06.184888 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:06.185520 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:06.185528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:06.185917 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 09:14:06.186216 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 09:14:06.186729 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":387,"threads_started":1}
I20260806 09:14:06.189450 24637 heartbeater.cc:347] Connected to a master server at 127.3.69.62:37087
I20260806 09:14:06.189781 24637 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:06.190498 24637 heartbeater.cc:511] Master 127.3.69.62:37087 requested a full tablet report, sending...
I20260806 09:14:06.192001 23960 ts_manager.cc:194] Registered new tserver with Master: 1ca9ab304e28471e9a9efd767fb2da75 (127.3.69.3:33247)
I20260806 09:14:06.192834 23960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:59153
I20260806 09:14:06.201166 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:14:06.210045 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.202328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.3.69.62:37087)
0806 09:14:06.202863 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.202869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.202888 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:14:06.203177 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.203186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.203380 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.203593 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:14:06.204168 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.205397 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.207793 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.208220 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.208372 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.208943 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:14:06.209398 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:14:06.209631 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.209690 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.209755 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393}
I20260806 09:14:06.211141 23960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:14:06.218227 23960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:14:06.242419 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.231945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.3.69.1:38897)
0806 09:14:06.234297 (+ 2352us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.234307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.234335 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:14:06.234435 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.234442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.234649 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.234856 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:06.236053 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.237209 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.239497 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.241021 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.241569 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.241794 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.241890 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.242050 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2141,"thread_start_us":1958,"threads_started":1}
I20260806 09:14:06.242482 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.230987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.3.69.2:38499)
0806 09:14:06.231786 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.231800 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.232447 (+ 647us) server_negotiation.cc:407] Connection header received
0806 09:14:06.233106 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.233118 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.233418 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.233632 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:06.234718 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.236341 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.239524 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.241028 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.241172 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.241400 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.241979 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.242168 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":442,"threads_started":1}
I20260806 09:14:06.242998 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.231223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51194)
0806 09:14:06.232604 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.232650 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:06.232827 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:06.235150 (+ 2323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:06.235157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:06.235190 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:06.235853 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.235872 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.237358 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.237365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.239293 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.239308 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.241303 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.241309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.241579 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.241686 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 09:14:06.241772 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 09:14:06.242024 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":102,"spinlock_wait_cycles":1408,"thread_start_us":1099,"threads_started":1}
I20260806 09:14:06.243383 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.230662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:38499 (local address 127.0.0.1:43468)
0806 09:14:06.232265 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.232329 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:06.232829 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:06.234017 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:06.234024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:06.234056 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:06.234490 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.234506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.236499 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.236504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.239382 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.239397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.241283 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.241290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.241567 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.241641 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 09:14:06.241837 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 09:14:06.242918 (+ 1081us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"mutex_wait_us":301,"spinlock_wait_cycles":4864,"thread_start_us":1169,"threads_started":1}
I20260806 09:14:06.243818 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.233162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.3.69.3:33247)
0806 09:14:06.234635 (+ 1473us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.234645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.234676 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:14:06.234793 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.234799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.235009 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.235230 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:06.236207 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.237848 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.240277 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.241286 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.241418 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.241657 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.242694 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.243112 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1294,"thread_start_us":1110,"threads_started":1}
I20260806 09:14:06.244066 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.232757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:33247 (local address 127.0.0.1:56146)
0806 09:14:06.233433 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.233454 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:06.233610 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:06.235513 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:06.235519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:06.235544 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:06.235998 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.236008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.238021 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.238026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.240117 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.240129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.241826 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.241833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.241996 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.242036 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 09:14:06.242121 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 09:14:06.242236 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":3,"thread_start_us":391,"threads_started":1}
I20260806 09:14:06.246009 24238 heartbeater.cc:503] Master 127.3.69.62:37087 was elected leader, sending a full tablet report...
I20260806 09:14:06.246785 24141 tablet_service.cc:1467] Processing CreateTablet for tablet 97224e57824840cfb16f27079958d0c2 (DEFAULT_TABLE table=test-table [id=dfaeb2e4cd034c33afc311fddbf19383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:06.248364 24141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97224e57824840cfb16f27079958d0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:06.249756 24346 tablet_service.cc:1467] Processing CreateTablet for tablet 97224e57824840cfb16f27079958d0c2 (DEFAULT_TABLE table=test-table [id=dfaeb2e4cd034c33afc311fddbf19383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:06.251219 24346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97224e57824840cfb16f27079958d0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:06.250876 24544 tablet_service.cc:1467] Processing CreateTablet for tablet 97224e57824840cfb16f27079958d0c2 (DEFAULT_TABLE table=test-table [id=dfaeb2e4cd034c33afc311fddbf19383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:06.252243 24544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97224e57824840cfb16f27079958d0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:06.260854 24659 tablet_bootstrap.cc:492] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Bootstrap starting.
I20260806 09:14:06.264505 24659 tablet_bootstrap.cc:654] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:06.265585 24659 log.cc:824] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:06.268273 24663 tablet_bootstrap.cc:492] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Bootstrap starting.
I20260806 09:14:06.268895 24659 tablet_bootstrap.cc:492] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: No bootstrap required, opened a new log
I20260806 09:14:06.269168 24659 ts_tablet_manager.cc:1397] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 09:14:06.271328 24663 tablet_bootstrap.cc:654] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:06.271494 24662 tablet_bootstrap.cc:492] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Bootstrap starting.
I20260806 09:14:06.272655 24663 log.cc:824] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:06.275452 24662 tablet_bootstrap.cc:654] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:06.275511 24663 tablet_bootstrap.cc:492] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: No bootstrap required, opened a new log
I20260806 09:14:06.275806 24663 ts_tablet_manager.cc:1397] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260806 09:14:06.276750 24662 log.cc:824] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:06.278955 24659 raft_consensus.cc:348] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.279505 24659 raft_consensus.cc:374] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:06.279665 24659 raft_consensus.cc:729] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33700e8d1a842d8ba5c6de6c7c07548, State: Initialized, Role: FOLLOWER
I20260806 09:14:06.280123 24662 tablet_bootstrap.cc:492] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: No bootstrap required, opened a new log
I20260806 09:14:06.280411 24662 ts_tablet_manager.cc:1397] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20260806 09:14:06.280335 24659 consensus_queue.cc:260] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.282220 24659 ts_tablet_manager.cc:1428] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260806 09:14:06.286129 24663 raft_consensus.cc:348] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.286612 24663 raft_consensus.cc:374] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:06.286726 24663 raft_consensus.cc:729] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca9ab304e28471e9a9efd767fb2da75, State: Initialized, Role: FOLLOWER
I20260806 09:14:06.287439 24663 consensus_queue.cc:260] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.288656 24637 heartbeater.cc:503] Master 127.3.69.62:37087 was elected leader, sending a full tablet report...
I20260806 09:14:06.289249 24663 ts_tablet_manager.cc:1428] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260806 09:14:06.289405 24662 raft_consensus.cc:348] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.289840 24662 raft_consensus.cc:374] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:06.289984 24662 raft_consensus.cc:729] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ddc95cfea24968a64cc07858ecd9ff, State: Initialized, Role: FOLLOWER
I20260806 09:14:06.290601 24662 consensus_queue.cc:260] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.291729 24448 heartbeater.cc:503] Master 127.3.69.62:37087 was elected leader, sending a full tablet report...
I20260806 09:14:06.292527 24662 ts_tablet_manager.cc:1428] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260806 09:14:06.408216 24669 raft_consensus.cc:482] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:14:06.408569 24669 raft_consensus.cc:504] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.410508 24669 leader_election.cc:288] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ddc95cfea24968a64cc07858ecd9ff (127.3.69.2:38499), b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897)
W20260806 09:14:06.415642 24638 tablet.cc:2369] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:14:06.419265 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.411125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:57413 (local address 127.3.69.2:38499)
0806 09:14:06.411448 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.411458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.412101 (+ 643us) server_negotiation.cc:407] Connection header received
0806 09:14:06.412210 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.412214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.412274 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.412346 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:06.412896 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.413945 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.416869 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.418141 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.418294 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.418396 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.418831 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.418970 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260806 09:14:06.419269 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.410993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:34497 (local address 127.3.69.1:38897)
0806 09:14:06.411270 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.411276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.411537 (+ 261us) server_negotiation.cc:407] Connection header received
0806 09:14:06.411777 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.411784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.411877 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.411965 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:06.412752 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.413946 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.416869 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.418141 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.418294 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.418400 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.418831 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.418970 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:14:06.419358 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.410941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.3.69.3:34497)
0806 09:14:06.411383 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.411399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:06.411590 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:06.412030 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:06.412036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:06.412078 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:06.412296 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.412305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.414106 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.414112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.416678 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.416694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.418336 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.418343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.418616 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.418669 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 09:14:06.418774 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 09:14:06.418976 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":275}
I20260806 09:14:06.419459 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.411026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:38499 (local address 127.3.69.3:57413)
0806 09:14:06.411966 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.411986 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:06.412099 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:06.412391 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:06.412398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:06.412428 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:06.412713 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.412722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.414106 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.414112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.416680 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:06.416694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.418362 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.418367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:06.418610 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.418668 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 09:14:06.418773 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 09:14:06.418976 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":171,"thread_start_us":492,"threads_started":1}
I20260806 09:14:06.420496 24173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97224e57824840cfb16f27079958d0c2" candidate_uuid: "1ca9ab304e28471e9a9efd767fb2da75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" is_pre_election: true
I20260806 09:14:06.420804 24375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97224e57824840cfb16f27079958d0c2" candidate_uuid: "1ca9ab304e28471e9a9efd767fb2da75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ddc95cfea24968a64cc07858ecd9ff" is_pre_election: true
I20260806 09:14:06.421065 24173 raft_consensus.cc:2454] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca9ab304e28471e9a9efd767fb2da75 in term 0.
I20260806 09:14:06.421437 24375 raft_consensus.cc:2454] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca9ab304e28471e9a9efd767fb2da75 in term 0.
I20260806 09:14:06.421902 24475 leader_election.cc:302] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca9ab304e28471e9a9efd767fb2da75, b33700e8d1a842d8ba5c6de6c7c07548; no voters:
I20260806 09:14:06.422374 24669 raft_consensus.cc:2781] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:14:06.422534 24669 raft_consensus.cc:482] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:14:06.422675 24669 raft_consensus.cc:3037] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:06.425350 24669 raft_consensus.cc:504] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.426121 24669 leader_election.cc:288] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [CANDIDATE]: Term 1 election: Requested vote from peers 08ddc95cfea24968a64cc07858ecd9ff (127.3.69.2:38499), b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897)
I20260806 09:14:06.426558 24375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97224e57824840cfb16f27079958d0c2" candidate_uuid: "1ca9ab304e28471e9a9efd767fb2da75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ddc95cfea24968a64cc07858ecd9ff"
I20260806 09:14:06.426611 24173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97224e57824840cfb16f27079958d0c2" candidate_uuid: "1ca9ab304e28471e9a9efd767fb2da75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b33700e8d1a842d8ba5c6de6c7c07548"
I20260806 09:14:06.426811 24173 raft_consensus.cc:3037] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:06.426805 24375 raft_consensus.cc:3037] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:06.430181 24375 raft_consensus.cc:2454] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca9ab304e28471e9a9efd767fb2da75 in term 1.
I20260806 09:14:06.430181 24173 raft_consensus.cc:2454] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca9ab304e28471e9a9efd767fb2da75 in term 1.
I20260806 09:14:06.430599 24475 leader_election.cc:302] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca9ab304e28471e9a9efd767fb2da75, b33700e8d1a842d8ba5c6de6c7c07548; no voters:
I20260806 09:14:06.430971 24669 raft_consensus.cc:2781] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:14:06.431459 24669 raft_consensus.cc:686] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 LEADER]: Becoming Leader. State: Replica: 1ca9ab304e28471e9a9efd767fb2da75, State: Running, Role: LEADER
I20260806 09:14:06.432011 24669 consensus_queue.cc:237] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } }
I20260806 09:14:06.436362 23960 catalog_manager.cc:5495] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ca9ab304e28471e9a9efd767fb2da75 (127.3.69.3). New cstate: current_term: 1 leader_uuid: "1ca9ab304e28471e9a9efd767fb2da75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca9ab304e28471e9a9efd767fb2da75" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 33247 } health_report { overall_health: HEALTHY } } }
W20260806 09:14:06.448763 24449 tablet.cc:2369] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:14:06.458763 24239 tablet.cc:2369] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:14:06.482585 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.477060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.3.69.3:33247)
0806 09:14:06.477267 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.477272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.477290 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:14:06.477427 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.477431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.477509 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.477663 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:14:06.478007 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.478761 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.480365 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.480826 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.480932 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.481443 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:14:06.481959 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:14:06.482079 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.482249 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.482322 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:14:06.490803 24173 raft_consensus.cc:1261] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 1 FOLLOWER]: Refusing update from remote peer 1ca9ab304e28471e9a9efd767fb2da75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:14:06.490803 24375 raft_consensus.cc:1261] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 FOLLOWER]: Refusing update from remote peer 1ca9ab304e28471e9a9efd767fb2da75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:14:06.491856 24675 consensus_queue.cc:1035] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b33700e8d1a842d8ba5c6de6c7c07548" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:14:06.492233 24669 consensus_queue.cc:1035] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ddc95cfea24968a64cc07858ecd9ff" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:14:06.504532 24679 mvcc.cc:205] Tried to move back new op lower bound from 7315487320016314368 to 7315487319789113344. Current Snapshot: MvccSnapshot[applied={T|T < 7315487320016314368}]
I20260806 09:14:06.516517 23959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:14:06.516844 23959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:14:06.520963 23959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Sending DeleteTablet for 3 replicas of tablet 97224e57824840cfb16f27079958d0c2
I20260806 09:14:06.522280 24346 tablet_service.cc:1514] Processing DeleteTablet for tablet 97224e57824840cfb16f27079958d0c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:06 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:43468
I20260806 09:14:06.522281 24141 tablet_service.cc:1514] Processing DeleteTablet for tablet 97224e57824840cfb16f27079958d0c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:06 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:51194
I20260806 09:14:06.523257 24692 tablet_replica.cc:330] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: stopping tablet replica
I20260806 09:14:06.523247 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 24020
I20260806 09:14:06.523322 24544 tablet_service.cc:1514] Processing DeleteTablet for tablet 97224e57824840cfb16f27079958d0c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:06 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:56146
I20260806 09:14:06.523938 24691 tablet_replica.cc:330] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: stopping tablet replica
I20260806 09:14:06.524435 24691 raft_consensus.cc:2229] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:14:06.524979 24692 raft_consensus.cc:2229] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:14:06.525584 24691 raft_consensus.cc:2258] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:06.526110 24692 raft_consensus.cc:2258] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:06.529251 24691 ts_tablet_manager.cc:1905] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:06.531035 24692 ts_tablet_manager.cc:1905] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:06.536697 24691 ts_tablet_manager.cc:1918] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:06.536960 24691 log.cc:1196] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/97224e57824840cfb16f27079958d0c2
I20260806 09:14:06.537628 24691 ts_tablet_manager.cc:1939] T 97224e57824840cfb16f27079958d0c2 P 08ddc95cfea24968a64cc07858ecd9ff: Deleting consensus metadata
I20260806 09:14:06.539619 23941 catalog_manager.cc:4841] TS 08ddc95cfea24968a64cc07858ecd9ff (127.3.69.2:38499): tablet 97224e57824840cfb16f27079958d0c2 (table test-table [id=dfaeb2e4cd034c33afc311fddbf19383]) successfully deleted
I20260806 09:14:06.543560 24692 ts_tablet_manager.cc:1918] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:06.543735 24692 log.cc:1196] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/97224e57824840cfb16f27079958d0c2
I20260806 09:14:06.545326 24693 tablet_replica.cc:330] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: stopping tablet replica
I20260806 09:14:06.545822 24693 raft_consensus.cc:2229] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:14:06.546597 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.525013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.3.69.1:38897)
0806 09:14:06.525236 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.525244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.525267 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:14:06.525340 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.525349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.525449 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.525566 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:06.528880 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.530239 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.532239 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.532920 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.533039 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.537233 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:06.537251 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:06.537274 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:06.537327 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:06.542321 (+ 4994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.543383 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.543394 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.543399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.543579 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.544153 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.544158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.544162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.544688 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:06.544999 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.545972 (+ 973us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.546199 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:14:06.546581 24693 raft_consensus.cc:2258] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:06.547348 24230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:06.548678 24693 ts_tablet_manager.cc:1905] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:06.555786 24693 ts_tablet_manager.cc:1918] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:06.555972 24693 log.cc:1196] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/97224e57824840cfb16f27079958d0c2
I20260806 09:14:06.556598 24693 ts_tablet_manager.cc:1939] T 97224e57824840cfb16f27079958d0c2 P 1ca9ab304e28471e9a9efd767fb2da75: Deleting consensus metadata
I20260806 09:14:06.558555 23942 catalog_manager.cc:4841] TS 1ca9ab304e28471e9a9efd767fb2da75 (127.3.69.3:33247): tablet 97224e57824840cfb16f27079958d0c2 (table test-table [id=dfaeb2e4cd034c33afc311fddbf19383]) successfully deleted
I20260806 09:14:06.643296 24692 ts_tablet_manager.cc:1939] T 97224e57824840cfb16f27079958d0c2 P b33700e8d1a842d8ba5c6de6c7c07548: Deleting consensus metadata
I20260806 09:14:06.644147 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 24020
W20260806 09:14:06.680616 23942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.69.1:38897 (error 108)
I20260806 09:14:06.681146 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 24244
I20260806 09:14:06.683790 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.683229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51214)
0806 09:14:06.683479 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.683616 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260806 09:14:06.684079 23942 catalog_manager.cc:4584] TS b33700e8d1a842d8ba5c6de6c7c07548 (127.3.69.1:38897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97224e57824840cfb16f27079958d0c2: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
I20260806 09:14:06.705268 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.682614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.3.69.2:38499)
0806 09:14:06.682831 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.682838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.682859 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:14:06.682925 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.682930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.683023 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.683241 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:06.685248 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.686440 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.688417 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.689061 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.689183 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.692733 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:06.692752 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:06.692772 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:06.692814 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:06.700966 (+ 8152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.701893 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.701903 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.701907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.702066 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.702533 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.702538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.702542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.703051 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:06.703375 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.704643 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.704889 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:14:06.706084 24439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:06.737977 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.737398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51230)
0806 09:14:06.737649 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.737781 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:14:06.739223 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.738779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51240)
0806 09:14:06.738946 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.739045 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 09:14:06.799916 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.799387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51254)
0806 09:14:06.799626 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.799735 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 09:14:06.801364 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.800885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51258)
0806 09:14:06.801123 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.801212 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260806 09:14:06.802815 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 24244
I20260806 09:14:06.838806 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 24451
I20260806 09:14:06.856297 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.839809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.3.69.3:33247)
0806 09:14:06.840029 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:06.840035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:06.840058 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:14:06.840239 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:06.840250 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:06.840352 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:06.840493 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:06.842000 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.843271 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.844573 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:06.845304 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:06.845438 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:06.848428 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:06.848449 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:06.848471 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:06.848522 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:06.853019 (+ 4497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.853792 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.853801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.853805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.854033 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:06.854432 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:06.854438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:06.854441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:06.854944 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:06.855331 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:06.855542 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 09:14:06.855807 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:14:06.857105 24631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:06.882403 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.881884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51266)
0806 09:14:06.882124 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.882223 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 09:14:06.883700 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:06.883275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51270)
0806 09:14:06.883481 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:06.883573 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 09:14:06.985654 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 24451
I20260806 09:14:07.012539 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 23910
I20260806 09:14:07.013875 24007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:07.131189 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:07.130394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38897 (local address 127.0.0.1:51272)
0806 09:14:07.130809 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:07.130898 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:38897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
I20260806 09:14:07.131413 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 23910
I20260806 09:14:07.161054 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37087
--webserver_interface=127.3.69.62
--webserver_port=42147
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:07.508757 24701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:07.509033 24701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:07.509199 24701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:07.516321 24701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:14:07.516424 24701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:07.516543 24701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:07.516638 24701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:14:07.516721 24701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:14:07.526506 24701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:37087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=42147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:07.527912 24701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:07.529256 24701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:07.541174 24714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:07.541249 24710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:07.541203 24707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:07.542503 24701 server_base.cc:1034] running on GCE node
I20260806 09:14:07.543375 24701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:07.544780 24701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:07.546025 24701 hybrid_clock.cc:648] HybridClock initialized: now 1786007647545982 us; error 47 us; skew 500 ppm
I20260806 09:14:07.546387 24701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:07 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007647, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:07.549678 24701 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:14:07.551354 24701 webserver.cc:460] Webserver started at http://127.3.69.62:42147/ using document root <none> and password file <none>
I20260806 09:14:07.552045 24701 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:07.552201 24701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:07.557547 24701 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 09:14:07.562664 24726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:07.564067 24701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:14:07.564266 24701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "5110eeb691c44b3d838a41d30ea3d266"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "3edaac89e9b290bae08f9a4e46417d6d"
server_key_iv: "d4a849065276ee6b30acee7a7749d505"
server_key_version: "encryptionkey@0"
I20260806 09:14:07.565272 24701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:07.583653 24701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:07.587982 24701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:07.588296 24701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:07.607949 24701 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:37087
I20260806 09:14:07.607977 24805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:37087 every 8 connection(s)
I20260806 09:14:07.609592 24701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:14:07.616712 24806 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:14:07.619372 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 24701
I20260806 09:14:07.620446 24806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Bootstrap starting.
I20260806 09:14:07.620460 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:38897
--local_ip_for_outbound_sockets=127.3.69.1
--tserver_master_addrs=127.3.69.62:37087
--webserver_port=34221
--webserver_interface=127.3.69.1
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:14:07.645183 24806 log.cc:824] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:07.661612 24806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:14:07.662158 24806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266: Bootstrap complete.
I20260806 09:14:07.671866 24806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:07.672730 24806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5110eeb691c44b3d838a41d30ea3d266, State: Initialized, Role: FOLLOWER
I20260806 09:14:07.673434 24806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:07.673751 24806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:14:07.673875 24806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:14:07.674075 24806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 2 FOLLOWER]: Advancing to term 3
I20260806 09:14:07.677896 24806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:07.678398 24806 leader_election.cc:302] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5110eeb691c44b3d838a41d30ea3d266; no voters:
I20260806 09:14:07.679105 24806 leader_election.cc:288] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 09:14:07.679265 24810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 09:14:07.681529 24806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:14:07.683820 24810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [term 3 LEADER]: Becoming Leader. State: Replica: 5110eeb691c44b3d838a41d30ea3d266, State: Running, Role: LEADER
I20260806 09:14:07.684561 24810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } }
I20260806 09:14:07.687580 24812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5110eeb691c44b3d838a41d30ea3d266. Latest consensus state: current_term: 3 leader_uuid: "5110eeb691c44b3d838a41d30ea3d266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } } }
I20260806 09:14:07.688004 24812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:07.688613 24811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5110eeb691c44b3d838a41d30ea3d266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5110eeb691c44b3d838a41d30ea3d266" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37087 } } }
I20260806 09:14:07.688939 24811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5110eeb691c44b3d838a41d30ea3d266 [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:07.691145 24823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:14:07.708659 24823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:14:07.710709 24823 catalog_manager.cc:1259] Loaded cluster ID: f38038106ddf441282b1b85c5ed53863
I20260806 09:14:07.710911 24823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 09:14:07.713814 24823 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f38982719ad google::LogMessage::Fail() at ??:0
@ 0x7f389827488a google::LogMessage::SendToLog() at ??:0
@ 0x7f38982714de google::LogMessage::Flush() at ??:0
@ 0x7f3898272a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f38adca4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f38adca1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f38add29610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f38add29439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f389aebd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f389aec091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f389aec0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f389ae9beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f389e1756db start_thread at ??:0
@ 0x7f389710b61f clone at ??:0
W20260806 09:14:08.015035 24808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:08.015451 24808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:08.015640 24808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:08.023041 24808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:08.023226 24808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:08.023424 24808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:14:08.033529 24808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:38897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=34221
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:08.034950 24808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:08.036309 24808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:08.049765 24865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:08.049791 24868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:08.049865 24866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:08.050685 24808 server_base.cc:1034] running on GCE node
I20260806 09:14:08.051404 24808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:08.052258 24808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:08.053519 24808 hybrid_clock.cc:648] HybridClock initialized: now 1786007648053433 us; error 77 us; skew 500 ppm
I20260806 09:14:08.053869 24808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:08 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007648, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:08.057091 24808 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:14:08.058573 24808 webserver.cc:460] Webserver started at http://127.3.69.1:34221/ using document root <none> and password file <none>
I20260806 09:14:08.059352 24808 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:08.059502 24808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:08.064908 24808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 09:14:08.067593 24880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:08.068840 24808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:08.069036 24808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "b33700e8d1a842d8ba5c6de6c7c07548"
format_stamp: "Formatted at 2026-08-06 09:14:01 on dist-test-slave-w4v5"
server_key: "a8804f1f593431f054557916c9c8cf1a"
server_key_iv: "cd5cd400567b6e2903a0e0cd4912683a"
server_key_version: "encryptionkey@0"
I20260806 09:14:08.070026 24808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:08.106828 24808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:08.110828 24808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:08.111131 24808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:08.112157 24808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:08.113556 24808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:08.113698 24808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:08.113831 24808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:08.113925 24808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:08.149037 24808 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:38897
I20260806 09:14:08.149130 25058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:38897 every 8 connection(s)
I20260806 09:14:08.150563 24808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:14:08.152045 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 24808
I20260806 09:14:08.152890 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:38499
--local_ip_for_outbound_sockets=127.3.69.2
--tserver_master_addrs=127.3.69.62:37087
--webserver_port=42201
--webserver_interface=127.3.69.2
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:14:08.156335 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:08.153931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:38643)
0806 09:14:08.154874 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:08.155171 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":350,"threads_started":1}
W20260806 09:14:08.157133 25059 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:37087 (0 consecutive failures): Network error: Failed to ping master at 127.3.69.62:37087: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
I20260806 09:14:08.158135 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:08.157693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:57911)
0806 09:14:08.157888 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:08.157984 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 09:14:08.159147 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:08.158780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:47091)
0806 09:14:08.158909 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:08.158984 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:14:08.159451 25059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:14:08.497470 25062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:08.497731 25062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:08.497920 25062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:08.505116 25062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:08.505245 25062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:08.505429 25062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:14:08.514824 25062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:38499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=42201
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:08.516341 25062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:08.517747 25062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:08.531032 25076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:08.530972 25078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:08.531067 25074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:08.531876 25062 server_base.cc:1034] running on GCE node
I20260806 09:14:08.532505 25062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:08.533488 25062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:08.534762 25062 hybrid_clock.cc:648] HybridClock initialized: now 1786007648534700 us; error 64 us; skew 500 ppm
I20260806 09:14:08.535131 25062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:08 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007648, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:08.538312 25062 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:14:08.539789 25062 webserver.cc:460] Webserver started at http://127.3.69.2:42201/ using document root <none> and password file <none>
I20260806 09:14:08.540438 25062 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:08.540546 25062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:08.545713 25062 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 09:14:08.548422 25091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:08.549749 25062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:08.549924 25062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "08ddc95cfea24968a64cc07858ecd9ff"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "df83fabbea96d081643ecb0ea1659600"
server_key_iv: "cc3daea686ed67803c6e7dd10264d060"
server_key_version: "encryptionkey@0"
I20260806 09:14:08.550951 25062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:08.567801 25062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:08.571957 25062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:08.572239 25062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:08.573290 25062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:08.574828 25062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:08.574950 25062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:08.575079 25062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:08.575192 25062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 09:14:08.610230 25062 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:38499
I20260806 09:14:08.610317 25253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:38499 every 8 connection(s)
I20260806 09:14:08.611900 25062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:14:08.612845 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 25062
I20260806 09:14:08.613646 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:33247
--local_ip_for_outbound_sockets=127.3.69.3
--tserver_master_addrs=127.3.69.62:37087
--webserver_port=34037
--webserver_interface=127.3.69.3
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:14:08.617180 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:08.615155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:33651)
0806 09:14:08.616022 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:08.616269 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":368,"threads_started":1}
W20260806 09:14:08.617950 25254 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:37087 (0 consecutive failures): Network error: Failed to ping master at 127.3.69.62:37087: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
I20260806 09:14:08.618784 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:08.618450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:52587)
0806 09:14:08.618602 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:08.618682 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 09:14:08.619793 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:08.619469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:40189)
0806 09:14:08.619597 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:08.619692 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 09:14:08.620121 25254 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:14:08.957636 25257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:08.957893 25257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:08.957998 25257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:08.964946 25257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:08.965052 25257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:08.965163 25257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:14:08.974502 25257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:33247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=34037
--tserver_master_addrs=127.3.69.62:37087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:08.975751 25257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:08.977046 25257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:08.990249 25264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:08.990216 25269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:08.990340 25265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:08.991595 25257 server_base.cc:1034] running on GCE node
I20260806 09:14:08.992271 25257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:08.993219 25257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:08.994459 25257 hybrid_clock.cc:648] HybridClock initialized: now 1786007648994409 us; error 60 us; skew 500 ppm
I20260806 09:14:08.994784 25257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:08 dist-test-slave-w4v5 krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007648, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:09.001714 25257 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:14:09.003217 25257 webserver.cc:460] Webserver started at http://127.3.69.3:34037/ using document root <none> and password file <none>
I20260806 09:14:09.003952 25257 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:09.004104 25257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:09.009663 25257 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 09:14:09.012581 25280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:09.013952 25257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:09.014140 25257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "1ca9ab304e28471e9a9efd767fb2da75"
format_stamp: "Formatted at 2026-08-06 09:14:02 on dist-test-slave-w4v5"
server_key: "8c1e3a7f7ad1ab4245e9af7b2cd11d93"
server_key_iv: "91dc3b7acb61888901560925b4683654"
server_key_version: "encryptionkey@0"
I20260806 09:14:09.015159 25257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:09.042567 25257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:09.046974 25257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:09.047287 25257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:09.048324 25257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:09.049747 25257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:09.049887 25257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:09.050032 25257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:09.050138 25257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:09.085323 25257 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:33247
I20260806 09:14:09.085415 25446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:33247 every 8 connection(s)
I20260806 09:14:09.086855 25257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:14:09.091979 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:09.090035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:42455)
0806 09:14:09.090848 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:09.091096 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":306,"threads_started":1}
W20260806 09:14:09.092839 25447 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:37087 (0 consecutive failures): Network error: Failed to ping master at 127.3.69.62:37087: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
I20260806 09:14:09.093161 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 25257
I20260806 09:14:09.093827 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:09.093401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:44953)
0806 09:14:09.093585 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:09.093685 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 09:14:09.094921 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:09.094567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:36057)
0806 09:14:09.094734 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:09.094812 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:14:09.095219 25447 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:14:09.160796 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:09.159976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:47055)
0806 09:14:09.160434 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:09.160562 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":218,"threads_started":1}
I20260806 09:14:09.621419 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:09.620630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:53351)
0806 09:14:09.621083 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:09.621170 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":217,"threads_started":1}
I20260806 09:14:10.096446 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:10.095684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:36247)
0806 09:14:10.096128 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:10.096214 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":214,"threads_started":1}
I20260806 09:14:10.162425 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:10.161739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:34115)
0806 09:14:10.162185 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:10.162267 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
I20260806 09:14:10.623051 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:10.622347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:42159)
0806 09:14:10.622805 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:10.622890 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":217,"threads_started":1}
I20260806 09:14:11.098268 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:11.097398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:49515)
0806 09:14:11.098009 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:11.098090 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":348,"threads_started":1}
I20260806 09:14:11.164260 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:11.163493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:50225)
0806 09:14:11.164008 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:11.164086 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":317,"threads_started":1}
I20260806 09:14:11.624848 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:11.624090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:57111)
0806 09:14:11.624575 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:11.624658 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
I20260806 09:14:12.100122 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:12.099375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:58827)
0806 09:14:12.099855 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:12.099944 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260806 09:14:12.165963 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:12.165192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:46129)
0806 09:14:12.165697 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:12.165778 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":248,"threads_started":1}
I20260806 09:14:12.626490 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:12.625791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:47969)
0806 09:14:12.626222 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:12.626300 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260806 09:14:13.101878 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:13.101148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:42301)
0806 09:14:13.101622 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:13.101705 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260806 09:14:13.167654 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:13.166920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:57593)
0806 09:14:13.167419 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:13.167493 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":305,"threads_started":1}
I20260806 09:14:13.628188 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:13.627463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:56407)
0806 09:14:13.627925 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:13.628018 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":230,"threads_started":1}
I20260806 09:14:14.103578 25471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:14.102766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:34389)
0806 09:14:14.103325 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:14.103405 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":268,"threads_started":1}
I20260806 09:14:14.169210 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:14.168501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:46969)
0806 09:14:14.168959 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:14.169044 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":249,"threads_started":1}
I20260806 09:14:14.629748 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:14.629053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:43365)
0806 09:14:14.629498 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:14.629587 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
I20260806 09:14:15.105340 25475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:15.104540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:40385)
0806 09:14:15.105047 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:15.105170 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260806 09:14:15.170964 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:15.170152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:50291)
0806 09:14:15.170648 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:15.170730 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":216,"threads_started":1}
I20260806 09:14:15.631665 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:15.630741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:50037)
0806 09:14:15.631354 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:15.631436 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":259,"threads_started":1}
I20260806 09:14:16.107837 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:16.106384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:43529)
0806 09:14:16.107527 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:16.107610 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":852,"threads_started":1}
I20260806 09:14:16.172662 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:16.171955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:52095)
0806 09:14:16.172418 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:16.172501 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":259,"threads_started":1}
I20260806 09:14:16.633457 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:16.632713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:36561)
0806 09:14:16.633198 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:16.633286 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":259,"threads_started":1}
I20260806 09:14:17.109452 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:17.108760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:46627)
0806 09:14:17.109202 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:17.109281 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
I20260806 09:14:17.174492 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:17.173754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:47991)
0806 09:14:17.174226 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:17.174315 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":224,"threads_started":1}
I20260806 09:14:17.635198 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:17.634358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:48845)
0806 09:14:17.634881 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:17.634958 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":236,"threads_started":1}
I20260806 09:14:18.111419 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:18.110443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:40625)
0806 09:14:18.111129 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:18.111228 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":344,"threads_started":1}
I20260806 09:14:18.176246 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:18.175494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:50269)
0806 09:14:18.175978 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:18.176065 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":273,"threads_started":1}
I20260806 09:14:18.636755 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:18.636032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:36821)
0806 09:14:18.636488 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:18.636571 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":252,"threads_started":1}
I20260806 09:14:19.113097 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:19.112341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:33581)
0806 09:14:19.112843 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:19.112926 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":293,"threads_started":1}
I20260806 09:14:19.177938 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:19.177206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:43637)
0806 09:14:19.177653 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:19.177757 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260806 09:14:19.638456 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:19.637676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:34951)
0806 09:14:19.638174 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:19.638267 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260806 09:14:20.114854 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:20.114110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:51099)
0806 09:14:20.114572 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:20.114663 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":242,"threads_started":1}
I20260806 09:14:20.179628 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:20.178874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:49945)
0806 09:14:20.179387 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:20.179469 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":293,"threads_started":1}
I20260806 09:14:20.640208 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:20.639391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:55411)
0806 09:14:20.639949 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:20.640044 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":287,"threads_started":1}
I20260806 09:14:21.116524 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:21.115787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:54591)
0806 09:14:21.116264 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:21.116358 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":230,"threads_started":1}
I20260806 09:14:21.181294 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:21.180522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:51793)
0806 09:14:21.180991 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:21.181090 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":263,"threads_started":1}
I20260806 09:14:21.641986 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:21.641172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:56413)
0806 09:14:21.641695 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:21.641785 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":239,"threads_started":1}
I20260806 09:14:22.118214 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:22.117438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:37653)
0806 09:14:22.117932 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:22.118018 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":260,"threads_started":1}
I20260806 09:14:22.182986 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:22.182229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:44181)
0806 09:14:22.182707 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:22.182792 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260806 09:14:22.643741 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:22.642961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:42109)
0806 09:14:22.643458 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:22.643539 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":280,"threads_started":1}
I20260806 09:14:23.119899 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:23.119191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:35001)
0806 09:14:23.119620 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:23.119707 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":215,"threads_started":1}
I20260806 09:14:23.185144 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:23.184248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:47361)
0806 09:14:23.184847 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:23.184947 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":277,"threads_started":1}
I20260806 09:14:23.645705 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:23.644888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:37433)
0806 09:14:23.645379 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:23.645469 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":263,"threads_started":1}
I20260806 09:14:24.105989 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 24808
I20260806 09:14:24.121892 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:24.121131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:40695)
0806 09:14:24.121602 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:24.121700 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":222,"threads_started":1}
I20260806 09:14:24.187222 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:24.186251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.1:54029)
0806 09:14:24.186793 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:24.186919 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":270,"threads_started":1}
I20260806 09:14:24.647691 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:24.646834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.2:50621)
0806 09:14:24.647430 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:24.647527 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":361,"threads_started":1}
I20260806 09:14:24.781656 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:24.107380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.3.69.1:38897)
0806 09:14:24.108049 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:24.108066 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:24.763994 (+655928us) server_negotiation.cc:407] Connection header received
0806 09:14:24.765795 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:24.765805 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:24.766247 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:24.766559 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:24.768271 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:24.769586 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:24.770888 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:24.771567 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:24.771723 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:24.775055 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:24.775063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:24.775124 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:24.775190 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:24.778208 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:24.778914 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:24.778922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:24.778928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:24.779146 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:24.779547 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:24.779552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:24.779556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:24.780095 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:24.780425 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:24.780902 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 09:14:24.781350 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":329,"threads_started":1}
I20260806 09:14:24.783211 25047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:24.869340 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 24808
I20260806 09:14:24.897245 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 25062
I20260806 09:14:24.915815 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:24.898364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.3.69.2:38499)
0806 09:14:24.899006 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:24.899021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:24.899063 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:14:24.900800 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:24.900809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:24.901241 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:24.901585 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:24.903240 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:24.904467 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:24.905726 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:24.906416 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:24.906620 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:24.909393 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:24.909401 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:24.909415 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:24.909452 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:24.912400 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:24.913109 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:24.913116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:24.913122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:24.913295 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:24.913703 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:24.913708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:24.913712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:24.914245 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:24.914556 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:24.915045 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 09:14:24.915496 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":308,"threads_started":1}
I20260806 09:14:24.917327 25249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:25.002017 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 25062
I20260806 09:14:25.031898 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 25257
I20260806 09:14:25.050156 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:25.032970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.3.69.3:33247)
0806 09:14:25.033627 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:25.033643 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:25.033682 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:14:25.035380 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:25.035392 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:25.035849 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:25.036150 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:25.037651 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:25.038922 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:25.040202 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:25.040875 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:25.041012 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:25.043803 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:25.043811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:25.043826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:25.043870 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:25.046945 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:25.047637 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:25.047643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:25.047650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:25.047843 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:25.048223 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:25.048226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:25.048228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:25.048749 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:25.049001 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:25.049411 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 09:14:25.049862 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":322,"threads_started":1}
I20260806 09:14:25.051811 25440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:25.137413 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:25.136209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:37087 (local address 127.3.69.3:47223)
0806 09:14:25.136815 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:25.137046 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":282,"threads_started":1}
I20260806 09:14:25.137450 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 25257
2026-08-06T09:14:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26000 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:14:25 dist-test-slave-w4v5 krb5kdc[25534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:14:25 dist-test-slave-w4v5 krb5kdc[25534](info): set up 2 sockets
Aug 06 09:14:25 dist-test-slave-w4v5 krb5kdc[25534](info): commencing operation
krb5kdc: starting...
W20260806 09:14:26.480798 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.006s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:14:26 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:14:26.783259 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45345
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:45345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:27.138717 25553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:27.138972 25553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:27.139154 25553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:27.146468 25553 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:14:27.146559 25553 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:14:27.146651 25553 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:14:27.146764 25553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:14:27.146868 25553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:27.146965 25553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:27.147044 25553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:14:27.147146 25553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:14:27.156988 25553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:45345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:27.158226 25553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:27.159618 25553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:27.171840 25560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:27.171795 25564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:27.172106 25559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:27.172989 25553 server_base.cc:1034] running on GCE node
I20260806 09:14:27.173780 25553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:27.175309 25553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:27.176580 25553 hybrid_clock.cc:648] HybridClock initialized: now 1786007667176513 us; error 39 us; skew 500 ppm
I20260806 09:14:27.176926 25553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:27 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007667, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:27.180209 25553 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:14:27.181784 25553 webserver.cc:460] Webserver started at http://127.3.69.62:42821/ using document root <none> and password file <none>
I20260806 09:14:27.182468 25553 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:27.182608 25553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:27.182927 25553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:14:27.185992 25553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "0b54eabcfc534e3598ac36fa9026c15b"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "40b943e94ec36e1d2079f361ecad61f5"
server_key_iv: "2375387cebe5272084fd7b9a27c732a0"
server_key_version: "encryptionkey@0"
I20260806 09:14:27.186790 25553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "0b54eabcfc534e3598ac36fa9026c15b"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "40b943e94ec36e1d2079f361ecad61f5"
server_key_iv: "2375387cebe5272084fd7b9a27c732a0"
server_key_version: "encryptionkey@0"
I20260806 09:14:27.191948 25553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 09:14:27.195832 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:27.197140 25553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:14:27.197340 25553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "0b54eabcfc534e3598ac36fa9026c15b"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "40b943e94ec36e1d2079f361ecad61f5"
server_key_iv: "2375387cebe5272084fd7b9a27c732a0"
server_key_version: "encryptionkey@0"
I20260806 09:14:27.197562 25553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:27.216526 25553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:27.220438 25553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:27.220729 25553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:27.238914 25553 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:45345
I20260806 09:14:27.238951 25657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:45345 every 8 connection(s)
I20260806 09:14:27.240372 25553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:14:27.243450 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 25553
I20260806 09:14:27.243673 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:14:27.244378 25658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:27.244493 3348 external_mini_cluster.cc:1428] Setting key 6a9369c364e944370a53d94bc6874bdf
I20260806 09:14:27.259438 25658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Bootstrap starting.
Aug 06 09:14:27 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:27.263518 25658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:27.264958 25658 log.cc:824] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:27.268635 25658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: No bootstrap required, opened a new log
I20260806 09:14:27.270530 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:27.246507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.3.69.62:45345)
0806 09:14:27.247465 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:27.247489 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:27.247548 (+ 59us) server_negotiation.cc:407] Connection header received
0806 09:14:27.249653 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:27.249683 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:27.250180 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:27.250672 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:27.252069 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:27.254099 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:27.255606 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:27.256733 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:27.256932 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:27.261409 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:27.261421 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:27.261440 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:27.261502 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:27.265629 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:27.266592 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:27.266603 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:27.266614 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:27.266822 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:27.267270 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:27.267275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:27.267278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:27.267990 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:27.268291 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:27.268675 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 09:14:27.269132 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":367,"threads_started":1}
I20260806 09:14:27.277545 25658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:14:27.277889 25658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:27.277954 25658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b54eabcfc534e3598ac36fa9026c15b, State: Initialized, Role: FOLLOWER
I20260806 09:14:27.278750 25658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:14:27.279119 25658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:14:27.279281 25658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:14:27.279464 25658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:27.282740 25658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:14:27.283391 25658 leader_election.cc:302] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b54eabcfc534e3598ac36fa9026c15b; no voters:
I20260806 09:14:27.286006 25658 leader_election.cc:288] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:14:27.286098 25664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:14:27.288007 25664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 1 LEADER]: Becoming Leader. State: Replica: 0b54eabcfc534e3598ac36fa9026c15b, State: Running, Role: LEADER
I20260806 09:14:27.288636 25664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:14:27.288982 25658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:14:27.292174 25665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b54eabcfc534e3598ac36fa9026c15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } } }
I20260806 09:14:27.292618 25665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:27.292148 25666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b54eabcfc534e3598ac36fa9026c15b. Latest consensus state: current_term: 1 leader_uuid: "0b54eabcfc534e3598ac36fa9026c15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } } }
I20260806 09:14:27.294028 25666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:27.295802 25681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:14:27.301371 25681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:14:27.310027 25681 catalog_manager.cc:1347] Generated new cluster ID: beaa5d7ade304422ab2113f2fe541288
I20260806 09:14:27.310138 25681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:14:27.324663 25681 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:14:27.326494 25681 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:14:27.338157 25681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Generated new TSK 0
I20260806 09:14:27.339227 25681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:14:27.424507 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45345
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:27.778970 25698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:27.779357 25698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:27.779533 25698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:27.787034 25698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:27.787228 25698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:27.787420 25698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:14:27.797857 25698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:27.799206 25698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:27.800530 25698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:27.814796 25706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:27.814792 25709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:27.814828 25712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:27.815625 25698 server_base.cc:1034] running on GCE node
I20260806 09:14:27.816340 25698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:27.817389 25698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:27.818638 25698 hybrid_clock.cc:648] HybridClock initialized: now 1786007667818581 us; error 47 us; skew 500 ppm
I20260806 09:14:27.818984 25698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:27 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007667, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:27.826592 25698 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:14:27.828290 25698 webserver.cc:460] Webserver started at http://127.3.69.1:45211/ using document root <none> and password file <none>
I20260806 09:14:27.829035 25698 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:27.829200 25698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:27.829532 25698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:14:27.832851 25698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "613ee6916e124eab9982c2633965dd09"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "c2cb1645ac098f3a13f521d00faf8b7b"
server_key_iv: "01c5bdd4c38e43dbe44221ee57772ecb"
server_key_version: "encryptionkey@0"
I20260806 09:14:27.833706 25698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "613ee6916e124eab9982c2633965dd09"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "c2cb1645ac098f3a13f521d00faf8b7b"
server_key_iv: "01c5bdd4c38e43dbe44221ee57772ecb"
server_key_version: "encryptionkey@0"
I20260806 09:14:27.839126 25698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 09:14:27.842633 25724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:27.843999 25698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:27.844201 25698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "613ee6916e124eab9982c2633965dd09"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "c2cb1645ac098f3a13f521d00faf8b7b"
server_key_iv: "01c5bdd4c38e43dbe44221ee57772ecb"
server_key_version: "encryptionkey@0"
I20260806 09:14:27.844398 25698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:27.859992 25698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:27.863891 25698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:27.864199 25698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:27.865207 25698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:27.866505 25698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:27.866580 25698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:27.866771 25698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:27.866825 25698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:27.902602 25698 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:46843
I20260806 09:14:27.902652 25893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:46843 every 8 connection(s)
I20260806 09:14:27.903949 25698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:14:27.906127 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 25698
I20260806 09:14:27.906373 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:14:27.907167 3348 external_mini_cluster.cc:1428] Setting key e8e13c6f8623a51039df0bfa2585a151
Aug 06 09:14:27 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007667, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:27.926681 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:27.907298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:57753 (local address 127.3.69.62:45345)
0806 09:14:27.908548 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:27.908554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:27.908574 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:14:27.910133 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:27.910137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:27.910213 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:27.910309 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:27.912516 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:27.913222 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:27.915663 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:27.916057 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:27.916159 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:27.920420 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:27.920424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:27.920427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:27.920459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:27.923412 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:27.924146 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:27.924154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:27.924161 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:27.924302 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:27.924710 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:27.924717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:27.924723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:27.925108 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:27.925279 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:27.925924 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 09:14:27.926184 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":1003,"threads_started":1}
I20260806 09:14:27.927217 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:27.906960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:57753)
0806 09:14:27.907934 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:27.907991 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:27.909830 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:27.910609 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:27.910622 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:27.911468 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:27.912312 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:27.912338 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:27.913389 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:27.913398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:27.915510 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:27.915528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:27.916283 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:27.916292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:27.916623 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:27.917354 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:27.917390 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:27.920173 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:27.923571 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:27.923578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:27.923589 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:27.924004 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:27.924401 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:27.924404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:27.924406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:27.924586 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:27.925232 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:27.925237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:27.925522 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 09:14:27.925742 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 09:14:27.926093 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":409,"threads_started":1}
I20260806 09:14:27.928721 25894 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45345
I20260806 09:14:27.929046 25894 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:27.929940 25894 heartbeater.cc:511] Master 127.3.69.62:45345 requested a full tablet report, sending...
I20260806 09:14:27.932312 25598 ts_manager.cc:194] Registered new tserver with Master: 613ee6916e124eab9982c2633965dd09 (127.3.69.1:46843)
I20260806 09:14:27.933758 25598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:57753
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:14:27.976291 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45345
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:28.323809 25902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:28.324095 25902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:28.324295 25902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:28.331535 25902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:28.331635 25902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:28.331833 25902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:14:28.341393 25902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:28.342736 25902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:28.344105 25902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:28.357990 25912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:28.357964 25916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:28.358111 25913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:28.358829 25902 server_base.cc:1034] running on GCE node
I20260806 09:14:28.359500 25902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:28.360477 25902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:28.361733 25902 hybrid_clock.cc:648] HybridClock initialized: now 1786007668361672 us; error 58 us; skew 500 ppm
I20260806 09:14:28.362102 25902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:28 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:28.365447 25902 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:14:28.366940 25902 webserver.cc:460] Webserver started at http://127.3.69.2:46471/ using document root <none> and password file <none>
I20260806 09:14:28.367651 25902 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:28.367759 25902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:28.368031 25902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:14:28.370918 25902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "d543852c35d1474691475d9bf606d5e0"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "cfe5f06b24bb48958ddf167cd6ee86f5"
server_key_iv: "6344a03a1b964e6927fec2afed7cb2ce"
server_key_version: "encryptionkey@0"
I20260806 09:14:28.371758 25902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "d543852c35d1474691475d9bf606d5e0"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "cfe5f06b24bb48958ddf167cd6ee86f5"
server_key_iv: "6344a03a1b964e6927fec2afed7cb2ce"
server_key_version: "encryptionkey@0"
I20260806 09:14:28.377048 25902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:14:28.380707 25929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:28.382031 25902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 09:14:28.382221 25902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "d543852c35d1474691475d9bf606d5e0"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "cfe5f06b24bb48958ddf167cd6ee86f5"
server_key_iv: "6344a03a1b964e6927fec2afed7cb2ce"
server_key_version: "encryptionkey@0"
I20260806 09:14:28.382424 25902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:28.397517 25902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:28.401659 25902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:28.401978 25902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:28.402997 25902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:28.404388 25902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:28.404495 25902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:28.404645 25902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:28.404726 25902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:28.442287 25902 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:41509
I20260806 09:14:28.442339 26097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:41509 every 8 connection(s)
I20260806 09:14:28.443727 25902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:14:28.444744 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 25902
I20260806 09:14:28.444959 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:14:28.445590 3348 external_mini_cluster.cc:1428] Setting key e5cfda410e9162bfa7f53c56fcc4acdf
Aug 06 09:14:28 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:28.468709 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:28.447004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:41305 (local address 127.3.69.62:45345)
0806 09:14:28.447572 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:28.447577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:28.448720 (+ 1143us) server_negotiation.cc:407] Connection header received
0806 09:14:28.451744 (+ 3024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:28.451748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:28.451823 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:28.451925 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:28.454110 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:28.454837 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:28.457425 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:28.457895 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:28.458005 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:28.462467 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:28.462472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:28.462475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:28.462508 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:28.465226 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:28.466225 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:28.466229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:28.466231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:28.466360 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:28.466871 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:28.466874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:28.466876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:28.467215 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:28.467333 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:28.468228 (+ 895us) server_negotiation.cc:299] Negotiation successful
0806 09:14:28.468400 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":337,"threads_started":1}
I20260806 09:14:28.470233 26103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:28.447512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:41305)
0806 09:14:28.448507 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:28.448604 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:28.451415 (+ 2811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:28.452202 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:28.452215 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:28.453048 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:28.453902 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:28.453931 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:28.455015 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:28.455023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:28.457252 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:28.457272 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:28.458141 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:28.458150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:28.458479 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:28.459330 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:28.459367 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:28.462214 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:28.465375 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:28.465387 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:28.465409 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:28.466079 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:28.466477 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:28.466484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:28.466491 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:28.466751 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:28.467349 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:28.467359 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:28.467802 (+ 443us) client_negotiation.cc:769] Sending connection context
0806 09:14:28.468118 (+ 316us) client_negotiation.cc:240] Negotiation successful
0806 09:14:28.468682 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":425,"threads_started":1}
I20260806 09:14:28.471822 26099 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45345
I20260806 09:14:28.472151 26099 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:28.472959 26099 heartbeater.cc:511] Master 127.3.69.62:45345 requested a full tablet report, sending...
I20260806 09:14:28.474534 25598 ts_manager.cc:194] Registered new tserver with Master: d543852c35d1474691475d9bf606d5e0 (127.3.69.2:41509)
I20260806 09:14:28.475623 25598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:41305
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:14:28.518726 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45345
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:28.863255 26109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:28.863610 26109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:28.863790 26109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:28.871212 26109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:28.871348 26109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:28.871538 26109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:14:28.881588 26109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:28.882874 26109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:28.884258 26109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:28.898070 26115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:28.898161 26118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:28.898132 26116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:28.899158 26109 server_base.cc:1034] running on GCE node
I20260806 09:14:28.899835 26109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:28.900710 26109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:28.901963 26109 hybrid_clock.cc:648] HybridClock initialized: now 1786007668901884 us; error 46 us; skew 500 ppm
I20260806 09:14:28.902321 26109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:28 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:28.905684 26109 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:14:28.907454 26109 webserver.cc:460] Webserver started at http://127.3.69.3:37485/ using document root <none> and password file <none>
I20260806 09:14:28.908160 26109 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:28.908300 26109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:28.908630 26109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:14:28.911669 26109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "75153d0835724ccbae2d4e66bc1e049b"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "0b794f8db22c0d8fa95c031578df7071"
server_key_iv: "a8bb0a92262e3af35e6a76cc5b17c880"
server_key_version: "encryptionkey@0"
I20260806 09:14:28.912422 26109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "75153d0835724ccbae2d4e66bc1e049b"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "0b794f8db22c0d8fa95c031578df7071"
server_key_iv: "a8bb0a92262e3af35e6a76cc5b17c880"
server_key_version: "encryptionkey@0"
I20260806 09:14:28.917542 26109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:14:28.921015 26131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:28.922318 26109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:14:28.922567 26109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "75153d0835724ccbae2d4e66bc1e049b"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "0b794f8db22c0d8fa95c031578df7071"
server_key_iv: "a8bb0a92262e3af35e6a76cc5b17c880"
server_key_version: "encryptionkey@0"
I20260806 09:14:28.922786 26109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:28.937485 25894 heartbeater.cc:503] Master 127.3.69.62:45345 was elected leader, sending a full tablet report...
I20260806 09:14:28.950105 26109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:28.954501 26109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:28.954815 26109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:28.955843 26109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:28.957197 26109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:28.957273 26109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:28.957357 26109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:28.957401 26109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:28.992650 26109 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:46699
I20260806 09:14:28.992741 26293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:46699 every 8 connection(s)
I20260806 09:14:28.993971 26109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:14:28.997825 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 26109
I20260806 09:14:28.998042 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:14:28.998693 3348 external_mini_cluster.cc:1428] Setting key 215365a7980627a58376293f52f55a5b
Aug 06 09:14:29 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:29.015811 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:28.996762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:40315 (local address 127.3.69.62:45345)
0806 09:14:28.997247 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:28.997252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:28.998284 (+ 1032us) server_negotiation.cc:407] Connection header received
0806 09:14:29.000879 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.000886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.001009 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.001138 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:29.003060 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.003908 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.005448 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.005852 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.005947 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.009471 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:29.009476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:29.009479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:29.009513 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:29.012342 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.013129 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.013134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.013137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.013302 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.013782 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.013787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.013791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.014167 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:29.014323 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.015343 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.015523 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":246,"threads_started":1}
I20260806 09:14:29.016824 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:28.997200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:40315)
0806 09:14:28.998059 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:28.998134 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:29.000464 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:29.001412 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:29.001425 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:29.002287 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:29.002881 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.002899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.004059 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.004063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.005315 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.005328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.006046 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.006051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.006262 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.006901 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:29.006924 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:29.009267 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:29.012533 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:29.012540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:29.012554 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:29.013016 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:29.013440 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:29.013447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:29.013455 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:29.013663 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:29.014364 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:29.014375 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:29.014870 (+ 495us) client_negotiation.cc:769] Sending connection context
0806 09:14:29.015248 (+ 378us) client_negotiation.cc:240] Negotiation successful
0806 09:14:29.015667 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":334,"threads_started":1}
I20260806 09:14:29.018239 26296 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45345
I20260806 09:14:29.018568 26296 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:29.019372 26296 heartbeater.cc:511] Master 127.3.69.62:45345 requested a full tablet report, sending...
I20260806 09:14:29.020839 25598 ts_manager.cc:194] Registered new tserver with Master: 75153d0835724ccbae2d4e66bc1e049b (127.3.69.3:46699)
I20260806 09:14:29.021733 25598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:40315
I20260806 09:14:29.027791 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:14:29.046452 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.032782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.3.69.62:45345)
0806 09:14:29.032997 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.033005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.033378 (+ 373us) server_negotiation.cc:407] Connection header received
0806 09:14:29.033580 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.033592 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.033677 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.033786 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:29.035370 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.036113 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.037506 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.037937 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.038100 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.040892 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:29.040897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:29.040901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:29.040936 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:29.043573 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.044265 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.044271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.044274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.044404 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.044883 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.044890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.044893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.045284 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:29.045557 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.045870 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.046127 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:14:29.051003 25598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:14:29.058207 25598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:14:29.081333 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.070796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:46699 (local address 127.0.0.1:46034)
0806 09:14:29.072178 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:29.072195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:29.072303 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:29.073727 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:29.073737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:29.073772 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:29.074311 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.074327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.076307 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.076313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.078469 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.078485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.080307 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.080315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.080545 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.080616 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 09:14:29.080721 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 09:14:29.080922 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"thread_start_us":424,"threads_started":1}
I20260806 09:14:29.081928 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.071510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.3.69.1:46843)
0806 09:14:29.072899 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.072908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.072938 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:14:29.073052 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.073067 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.073459 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.073679 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:29.074854 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.076309 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.078953 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.080422 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.080544 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.080796 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.081340 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.081529 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":2071,"threads_started":1}
I20260806 09:14:29.082398 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.070768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46843 (local address 127.0.0.1:51990)
0806 09:14:29.071585 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:29.071629 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:29.071804 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:29.073986 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:29.073990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:29.074011 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:29.074611 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.074629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.076491 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.076535 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.078785 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.078800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.080935 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.080941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.081131 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.081164 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:14:29.081263 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:14:29.081408 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":63,"thread_start_us":1149,"threads_started":1}
I20260806 09:14:29.082525 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.070912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.3.69.3:46699)
0806 09:14:29.072965 (+ 2053us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.072975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.073004 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:14:29.073284 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.073293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.073475 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.074327 (+ 852us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:29.074653 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.076139 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.078642 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.080064 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.081686 (+ 1622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.081922 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.082001 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.082155 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1900,"thread_start_us":1691,"threads_started":1}
I20260806 09:14:29.082836 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.072891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:41509 (local address 127.0.0.1:53274)
0806 09:14:29.073449 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:29.073466 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:29.073582 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:29.075478 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:29.075482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:29.075509 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:29.075878 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.075890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.077911 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.077917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.079914 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.079931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.081853 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.081859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.082063 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.082214 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 09:14:29.082327 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 09:14:29.082482 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":883,"threads_started":1}
I20260806 09:14:29.083359 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.073454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.3.69.2:41509)
0806 09:14:29.074884 (+ 1430us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.074894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.074920 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:14:29.075012 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.075021 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.075272 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.075460 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:29.076047 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.077761 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.080126 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.081391 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.081523 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.081702 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.082877 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.083036 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1265,"thread_start_us":1156,"threads_started":1}
I20260806 09:14:29.086354 26198 tablet_service.cc:1467] Processing CreateTablet for tablet 001026e43f2c4bb7b0215b8ee4d9d15a (DEFAULT_TABLE table=test-table [id=d5c25698142044f4862c3cd385ad9a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:29.086390 25800 tablet_service.cc:1467] Processing CreateTablet for tablet 001026e43f2c4bb7b0215b8ee4d9d15a (DEFAULT_TABLE table=test-table [id=d5c25698142044f4862c3cd385ad9a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:29.086637 26004 tablet_service.cc:1467] Processing CreateTablet for tablet 001026e43f2c4bb7b0215b8ee4d9d15a (DEFAULT_TABLE table=test-table [id=d5c25698142044f4862c3cd385ad9a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:29.087736 26198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001026e43f2c4bb7b0215b8ee4d9d15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:29.087839 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001026e43f2c4bb7b0215b8ee4d9d15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:29.087855 26004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 001026e43f2c4bb7b0215b8ee4d9d15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:29.099761 26324 tablet_bootstrap.cc:492] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Bootstrap starting.
I20260806 09:14:29.102448 26325 tablet_bootstrap.cc:492] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Bootstrap starting.
I20260806 09:14:29.103296 26326 tablet_bootstrap.cc:492] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Bootstrap starting.
I20260806 09:14:29.105000 26324 tablet_bootstrap.cc:654] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:29.106057 26326 tablet_bootstrap.cc:654] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:29.106194 26325 tablet_bootstrap.cc:654] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:29.106338 26324 log.cc:824] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:29.107265 26326 log.cc:824] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:29.107522 26325 log.cc:824] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:29.109530 26324 tablet_bootstrap.cc:492] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: No bootstrap required, opened a new log
I20260806 09:14:29.109799 26324 ts_tablet_manager.cc:1397] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 09:14:29.110298 26326 tablet_bootstrap.cc:492] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: No bootstrap required, opened a new log
I20260806 09:14:29.110579 26326 ts_tablet_manager.cc:1397] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 09:14:29.110667 26325 tablet_bootstrap.cc:492] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: No bootstrap required, opened a new log
I20260806 09:14:29.110975 26325 ts_tablet_manager.cc:1397] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 09:14:29.117246 26324 raft_consensus.cc:348] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.117617 26324 raft_consensus.cc:374] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:29.117756 26324 raft_consensus.cc:729] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75153d0835724ccbae2d4e66bc1e049b, State: Initialized, Role: FOLLOWER
I20260806 09:14:29.118384 26324 consensus_queue.cc:260] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.119524 26296 heartbeater.cc:503] Master 127.3.69.62:45345 was elected leader, sending a full tablet report...
I20260806 09:14:29.121430 26324 ts_tablet_manager.cc:1428] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260806 09:14:29.121330 26326 raft_consensus.cc:348] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.121797 26326 raft_consensus.cc:374] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:29.121923 26326 raft_consensus.cc:729] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613ee6916e124eab9982c2633965dd09, State: Initialized, Role: FOLLOWER
I20260806 09:14:29.122380 26325 raft_consensus.cc:348] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.122900 26325 raft_consensus.cc:374] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:29.123055 26325 raft_consensus.cc:729] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d543852c35d1474691475d9bf606d5e0, State: Initialized, Role: FOLLOWER
I20260806 09:14:29.123201 26326 consensus_queue.cc:260] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.123859 26325 consensus_queue.cc:260] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.125022 26326 ts_tablet_manager.cc:1428] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260806 09:14:29.126699 26099 heartbeater.cc:503] Master 127.3.69.62:45345 was elected leader, sending a full tablet report...
I20260806 09:14:29.127998 26325 ts_tablet_manager.cc:1428] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
W20260806 09:14:29.156432 25895 tablet.cc:2369] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:14:29.196151 26101 tablet.cc:2369] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:14:29.240295 26333 raft_consensus.cc:482] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:14:29.240633 26333 raft_consensus.cc:504] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.242782 26333 leader_election.cc:288] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75153d0835724ccbae2d4e66bc1e049b (127.3.69.3:46699), d543852c35d1474691475d9bf606d5e0 (127.3.69.2:41509)
W20260806 09:14:29.245862 26297 tablet.cc:2369] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:14:29.251708 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.243445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:39405 (local address 127.3.69.2:41509)
0806 09:14:29.243650 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.243656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.244445 (+ 789us) server_negotiation.cc:407] Connection header received
0806 09:14:29.244684 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.244690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.244778 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.244865 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:29.245700 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.246767 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.249380 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.250644 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.250794 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.250904 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.251290 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.251438 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:14:29.251941 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.243368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:57881 (local address 127.3.69.3:46699)
0806 09:14:29.243600 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.243609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.244324 (+ 715us) server_negotiation.cc:407] Connection header received
0806 09:14:29.244594 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.244598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.244702 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.244804 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:29.245504 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.246767 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.249680 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.250935 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.251115 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.251232 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.251554 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.251684 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:14:29.252072 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.243093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:46699 (local address 127.3.69.1:57881)
0806 09:14:29.244185 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:29.244222 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:29.244430 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:29.244831 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:29.244836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:29.244870 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:29.245319 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.245335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.246917 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.246922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.249500 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.249514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.251197 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.251203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.251379 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.251403 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:14:29.251515 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:14:29.251650 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":161,"thread_start_us":698,"threads_started":1}
I20260806 09:14:29.252733 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.243093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:41509 (local address 127.3.69.1:39405)
0806 09:14:29.244323 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:29.244341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:29.244475 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:29.245026 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:29.245030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:29.245051 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:29.245485 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.245498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.246983 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.246986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.249171 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:29.249185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.250875 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.250882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:29.251049 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.251115 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 09:14:29.252224 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0806 09:14:29.252352 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":416,"thread_start_us":580,"threads_started":1}
I20260806 09:14:29.253149 26220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001026e43f2c4bb7b0215b8ee4d9d15a" candidate_uuid: "613ee6916e124eab9982c2633965dd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75153d0835724ccbae2d4e66bc1e049b" is_pre_election: true
I20260806 09:14:29.253751 26220 raft_consensus.cc:2454] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613ee6916e124eab9982c2633965dd09 in term 0.
I20260806 09:14:29.254042 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001026e43f2c4bb7b0215b8ee4d9d15a" candidate_uuid: "613ee6916e124eab9982c2633965dd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d543852c35d1474691475d9bf606d5e0" is_pre_election: true
I20260806 09:14:29.254488 25729 leader_election.cc:302] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613ee6916e124eab9982c2633965dd09, 75153d0835724ccbae2d4e66bc1e049b; no voters:
I20260806 09:14:29.254601 26035 raft_consensus.cc:2454] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613ee6916e124eab9982c2633965dd09 in term 0.
I20260806 09:14:29.254910 26333 raft_consensus.cc:2781] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:14:29.255115 26333 raft_consensus.cc:482] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:14:29.255226 26333 raft_consensus.cc:3037] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:29.257576 26333 raft_consensus.cc:504] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.258301 26333 leader_election.cc:288] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [CANDIDATE]: Term 1 election: Requested vote from peers 75153d0835724ccbae2d4e66bc1e049b (127.3.69.3:46699), d543852c35d1474691475d9bf606d5e0 (127.3.69.2:41509)
I20260806 09:14:29.258701 26220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001026e43f2c4bb7b0215b8ee4d9d15a" candidate_uuid: "613ee6916e124eab9982c2633965dd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75153d0835724ccbae2d4e66bc1e049b"
I20260806 09:14:29.258834 26035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "001026e43f2c4bb7b0215b8ee4d9d15a" candidate_uuid: "613ee6916e124eab9982c2633965dd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d543852c35d1474691475d9bf606d5e0"
I20260806 09:14:29.258924 26220 raft_consensus.cc:3037] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:29.259112 26035 raft_consensus.cc:3037] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:29.261236 26220 raft_consensus.cc:2454] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613ee6916e124eab9982c2633965dd09 in term 1.
I20260806 09:14:29.261406 26035 raft_consensus.cc:2454] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613ee6916e124eab9982c2633965dd09 in term 1.
I20260806 09:14:29.261688 25729 leader_election.cc:302] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613ee6916e124eab9982c2633965dd09, 75153d0835724ccbae2d4e66bc1e049b; no voters:
I20260806 09:14:29.261986 26333 raft_consensus.cc:2781] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:14:29.262483 26333 raft_consensus.cc:686] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 1 LEADER]: Becoming Leader. State: Replica: 613ee6916e124eab9982c2633965dd09, State: Running, Role: LEADER
I20260806 09:14:29.263087 26333 consensus_queue.cc:237] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:29.268239 25596 catalog_manager.cc:5495] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 613ee6916e124eab9982c2633965dd09 (127.3.69.1). New cstate: current_term: 1 leader_uuid: "613ee6916e124eab9982c2633965dd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } health_report { overall_health: UNKNOWN } } }
I20260806 09:14:29.323614 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.317827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.3.69.1:46843)
0806 09:14:29.318069 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.318076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.318101 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:14:29.318216 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.318220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.318319 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.318486 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:14:29.318916 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.319710 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.321282 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.321697 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.321835 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.322371 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:14:29.322908 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:14:29.323123 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.323248 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.323336 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:14:29.332777 26035 raft_consensus.cc:1261] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Refusing update from remote peer 613ee6916e124eab9982c2633965dd09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:14:29.332794 26220 raft_consensus.cc:1261] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 1 FOLLOWER]: Refusing update from remote peer 613ee6916e124eab9982c2633965dd09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:14:29.333842 26333 consensus_queue.cc:1035] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:14:29.334142 26341 consensus_queue.cc:1035] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:14:29.344601 26346 mvcc.cc:205] Tried to move back new op lower bound from 7315487413575868416 to 7315487413306216448. Current Snapshot: MvccSnapshot[applied={T|T < 7315487413575868416}]
I20260806 09:14:29.357915 25598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:14:29.358240 25598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:14:29.362039 25598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Sending DeleteTablet for 3 replicas of tablet 001026e43f2c4bb7b0215b8ee4d9d15a
I20260806 09:14:29.363322 26198 tablet_service.cc:1514] Processing DeleteTablet for tablet 001026e43f2c4bb7b0215b8ee4d9d15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:29 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:46034
I20260806 09:14:29.363497 26004 tablet_service.cc:1514] Processing DeleteTablet for tablet 001026e43f2c4bb7b0215b8ee4d9d15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:29 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:53274
I20260806 09:14:29.363700 25800 tablet_service.cc:1514] Processing DeleteTablet for tablet 001026e43f2c4bb7b0215b8ee4d9d15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:29 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:51990
I20260806 09:14:29.364528 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 25698
I20260806 09:14:29.375929 26360 tablet_replica.cc:330] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: stopping tablet replica
I20260806 09:14:29.376696 26361 tablet_replica.cc:330] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: stopping tablet replica
I20260806 09:14:29.377367 26360 raft_consensus.cc:2229] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:14:29.377897 26359 tablet_replica.cc:330] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: stopping tablet replica
I20260806 09:14:29.377892 26361 raft_consensus.cc:2229] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:14:29.378379 26359 raft_consensus.cc:2229] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:14:29.378997 26359 raft_consensus.cc:2258] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:29.379029 26360 raft_consensus.cc:2258] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:29.379438 26361 raft_consensus.cc:2258] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:29.381639 26359 ts_tablet_manager.cc:1905] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:29.382395 26360 ts_tablet_manager.cc:1905] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:29.386082 26361 ts_tablet_manager.cc:1905] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:29.389709 26359 ts_tablet_manager.cc:1918] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:29.389927 26359 log.cc:1196] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/001026e43f2c4bb7b0215b8ee4d9d15a
I20260806 09:14:29.390316 26360 ts_tablet_manager.cc:1918] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:29.390573 26359 ts_tablet_manager.cc:1939] T 001026e43f2c4bb7b0215b8ee4d9d15a P 75153d0835724ccbae2d4e66bc1e049b: Deleting consensus metadata
I20260806 09:14:29.390614 26360 log.cc:1196] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/001026e43f2c4bb7b0215b8ee4d9d15a
I20260806 09:14:29.391324 26360 ts_tablet_manager.cc:1939] T 001026e43f2c4bb7b0215b8ee4d9d15a P d543852c35d1474691475d9bf606d5e0: Deleting consensus metadata
I20260806 09:14:29.392704 25577 catalog_manager.cc:4841] TS 75153d0835724ccbae2d4e66bc1e049b (127.3.69.3:46699): tablet 001026e43f2c4bb7b0215b8ee4d9d15a (table test-table [id=d5c25698142044f4862c3cd385ad9a8e]) successfully deleted
I20260806 09:14:29.393090 25578 catalog_manager.cc:4841] TS d543852c35d1474691475d9bf606d5e0 (127.3.69.2:41509): tablet 001026e43f2c4bb7b0215b8ee4d9d15a (table test-table [id=d5c25698142044f4862c3cd385ad9a8e]) successfully deleted
I20260806 09:14:29.396345 26361 ts_tablet_manager.cc:1918] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:29.396641 26361 log.cc:1196] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/001026e43f2c4bb7b0215b8ee4d9d15a
Aug 06 09:14:29 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:14:29.397295 26361 ts_tablet_manager.cc:1939] T 001026e43f2c4bb7b0215b8ee4d9d15a P 613ee6916e124eab9982c2633965dd09: Deleting consensus metadata
I20260806 09:14:29.399042 25578 catalog_manager.cc:4841] TS 613ee6916e124eab9982c2633965dd09 (127.3.69.1:46843): tablet 001026e43f2c4bb7b0215b8ee4d9d15a (table test-table [id=d5c25698142044f4862c3cd385ad9a8e]) successfully deleted
I20260806 09:14:29.406111 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.365910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.3.69.1:46843)
0806 09:14:29.366140 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.366150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.377057 (+ 10907us) server_negotiation.cc:407] Connection header received
0806 09:14:29.377457 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.377464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.377571 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.377698 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:29.380379 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.381708 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.388875 (+ 7167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.389651 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.389765 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.398385 (+ 8620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:29.398409 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:29.398428 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:29.398495 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:29.402700 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.403687 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.403697 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.403700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.403889 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.404370 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.404376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.404379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.404878 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:29.405202 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.405460 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.405714 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:14:29.406904 25888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:29.520328 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 25698
I20260806 09:14:29.547878 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 25902
Aug 06 09:14:29 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:14:29.563732 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.548649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.3.69.2:41509)
0806 09:14:29.548858 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.548863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.548884 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:14:29.549061 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.549069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.549166 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.549278 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:29.550622 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.551561 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.552882 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.553275 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.553411 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.557725 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:29.557741 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:29.557757 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:29.557792 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:29.560698 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.561425 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.561433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.561435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.561566 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.561978 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.561983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.561986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.562401 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:29.562794 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.563131 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.563392 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:14:29.564424 26092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:29.655499 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 25902
I20260806 09:14:29.682519 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 26109
Aug 06 09:14:29 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:14:29.699900 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:29.683407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.3.69.3:46699)
0806 09:14:29.683669 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:29.683675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:29.683697 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:14:29.683844 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:29.683849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:29.683936 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:29.684036 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:29.685432 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.686351 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.687866 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:29.688315 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:29.688423 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:29.693371 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:29.693397 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:29.693426 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:29.693472 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:29.696909 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.697735 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.697743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.697746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.697921 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:29.698329 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:29.698333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:29.698335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:29.698714 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:29.698939 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:29.699453 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 09:14:29.699635 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260806 09:14:29.700565 26288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:29.792331 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 26109
I20260806 09:14:29.817184 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 25553
I20260806 09:14:29.818534 25651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:29.932554 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 25553
I20260806 09:14:49.961334 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45345
--webserver_interface=127.3.69.62
--webserver_port=42821
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:45345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:14:50.302913 26369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:50.303264 26369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:50.303365 26369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:50.310360 26369 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:14:50.310443 26369 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:14:50.310498 26369 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:14:50.310554 26369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:14:50.310606 26369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:50.310681 26369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:50.310729 26369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:14:50.310775 26369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:14:50.320465 26369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:45345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:50.321656 26369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:50.322876 26369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:50.349123 26381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:50.349102 26384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:50.349210 26382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:50.349509 26369 server_base.cc:1034] running on GCE node
I20260806 09:14:50.350746 26369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:50.352330 26369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:50.353580 26369 hybrid_clock.cc:648] HybridClock initialized: now 1786007690353513 us; error 46 us; skew 500 ppm
I20260806 09:14:50.353982 26369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:50 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007690, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:50.357614 26369 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:14:50.359431 26369 webserver.cc:460] Webserver started at http://127.3.69.62:42821/ using document root <none> and password file <none>
I20260806 09:14:50.360209 26369 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:50.360363 26369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:50.366587 26369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 09:14:50.370244 26392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:50.371747 26369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:14:50.371953 26369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "0b54eabcfc534e3598ac36fa9026c15b"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "40b943e94ec36e1d2079f361ecad61f5"
server_key_iv: "2375387cebe5272084fd7b9a27c732a0"
server_key_version: "encryptionkey@0"
I20260806 09:14:50.373107 26369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:50.384964 26369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:50.389398 26369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:50.389706 26369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:50.407956 26369 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:45345
I20260806 09:14:50.407990 26475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:45345 every 8 connection(s)
I20260806 09:14:50.409454 26369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:14:50.411942 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 26369
I20260806 09:14:50.412854 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:46843
--local_ip_for_outbound_sockets=127.3.69.1
--tserver_master_addrs=127.3.69.62:45345
--webserver_port=45211
--webserver_interface=127.3.69.1
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:14:50.416694 26476 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:14:50.419986 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Bootstrap starting.
I20260806 09:14:50.446372 26476 log.cc:824] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:50.462597 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:14:50.463332 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Bootstrap complete.
I20260806 09:14:50.476963 26476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:14:50.478291 26476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b54eabcfc534e3598ac36fa9026c15b, State: Initialized, Role: FOLLOWER
I20260806 09:14:50.479043 26476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:14:50.479380 26476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:14:50.479537 26476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:14:50.479734 26476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:14:50.485626 26476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:14:50.486228 26476 leader_election.cc:302] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b54eabcfc534e3598ac36fa9026c15b; no voters:
I20260806 09:14:50.486897 26476 leader_election.cc:288] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:14:50.487041 26480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:14:50.488451 26480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 2 LEADER]: Becoming Leader. State: Replica: 0b54eabcfc534e3598ac36fa9026c15b, State: Running, Role: LEADER
I20260806 09:14:50.489013 26480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:14:50.489415 26476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:14:50.491531 26482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b54eabcfc534e3598ac36fa9026c15b. Latest consensus state: current_term: 2 leader_uuid: "0b54eabcfc534e3598ac36fa9026c15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } } }
I20260806 09:14:50.491920 26482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:50.492482 26481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b54eabcfc534e3598ac36fa9026c15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } } }
I20260806 09:14:50.492796 26481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: This master's current role is: LEADER
I20260806 09:14:50.501253 26486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:14:50.509541 26486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:14:50.511023 26486 catalog_manager.cc:1259] Loaded cluster ID: beaa5d7ade304422ab2113f2fe541288
I20260806 09:14:50.511209 26486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:14:50.515358 26486 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:14:50.517004 26486 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Loaded TSK: 0
I20260806 09:14:50.560770 26510 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:14:50.566601 26486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Generated new TSK 1
I20260806 09:14:50.567037 26486 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:14:50.819478 26478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:50.819940 26478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:50.820137 26478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:50.832329 26478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:50.832486 26478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:50.832686 26478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:14:50.848877 26478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:46843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=45211
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:50.850672 26478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:50.852584 26478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:50.868083 26521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:50.868070 26525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:50.868175 26518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:50.869696 26478 server_base.cc:1034] running on GCE node
I20260806 09:14:50.870466 26478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:50.871665 26478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:50.872961 26478 hybrid_clock.cc:648] HybridClock initialized: now 1786007690872878 us; error 80 us; skew 500 ppm
I20260806 09:14:50.873368 26478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:50 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007690, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:50.880707 26478 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:14:50.882524 26478 webserver.cc:460] Webserver started at http://127.3.69.1:45211/ using document root <none> and password file <none>
I20260806 09:14:50.883582 26478 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:50.883751 26478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:50.890820 26478 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260806 09:14:50.894109 26538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:50.895678 26478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:50.895852 26478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "613ee6916e124eab9982c2633965dd09"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "c2cb1645ac098f3a13f521d00faf8b7b"
server_key_iv: "01c5bdd4c38e43dbe44221ee57772ecb"
server_key_version: "encryptionkey@0"
I20260806 09:14:50.897138 26478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:50.922685 26478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:50.926769 26478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:50.927004 26478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:50.928025 26478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:50.929400 26478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:50.929522 26478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:50.929656 26478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:50.929742 26478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:50.965303 26478 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:46843
I20260806 09:14:50.965394 26701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:46843 every 8 connection(s)
I20260806 09:14:50.966799 26478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:14:50.971001 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 26478
I20260806 09:14:50.971873 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:41509
--local_ip_for_outbound_sockets=127.3.69.2
--tserver_master_addrs=127.3.69.62:45345
--webserver_port=46471
--webserver_interface=127.3.69.2
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:14:50 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007690, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:50.993917 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:50.969719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:48037 (local address 127.3.69.62:45345)
0806 09:14:50.970324 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:50.970342 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:50.971261 (+ 919us) server_negotiation.cc:407] Connection header received
0806 09:14:50.974381 (+ 3120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:50.974394 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:50.974725 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:50.975019 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:50.977122 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:50.978574 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:50.981076 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:50.981703 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:50.981836 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:50.986227 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:50.986235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:50.986249 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:50.986286 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:50.989270 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:50.990061 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:50.990068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:50.990074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:50.990241 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:50.990738 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:50.990742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:50.990744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:50.991281 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:50.991484 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:50.992264 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 09:14:50.992656 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":240,"threads_started":1}
I20260806 09:14:50.993917 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:50.969987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:48037)
0806 09:14:50.971043 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:50.971118 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:50.973849 (+ 2731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:50.975252 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:50.975265 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:50.976073 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:50.976839 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:50.976867 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:50.978748 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:50.978756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:50.980911 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:50.980930 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:50.981936 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:50.981945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:50.982273 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:50.983027 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:50.983062 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:50.985907 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:50.989409 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:50.989418 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:50.989433 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:50.989912 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:50.990379 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:50.990384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:50.990391 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:50.990608 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:50.991426 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:50.991434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:50.991768 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 09:14:50.992029 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 09:14:50.992569 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":467,"threads_started":1}
I20260806 09:14:50.997313 26702 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45345
I20260806 09:14:50.997635 26702 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:50.998364 26702 heartbeater.cc:511] Master 127.3.69.62:45345 requested a full tablet report, sending...
I20260806 09:14:51.001010 26419 ts_manager.cc:194] Registered new tserver with Master: 613ee6916e124eab9982c2633965dd09 (127.3.69.1:46843)
I20260806 09:14:51.003369 26419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:48037
W20260806 09:14:51.324216 26710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:51.324482 26710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:51.324584 26710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:51.331750 26710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:51.331851 26710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:51.331960 26710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:14:51.341409 26710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:41509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=46471
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:51.342572 26710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:51.343873 26710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:51.357163 26719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:51.357205 26722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:51.357245 26717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:51.358058 26710 server_base.cc:1034] running on GCE node
I20260806 09:14:51.358668 26710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:51.359635 26710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:51.360874 26710 hybrid_clock.cc:648] HybridClock initialized: now 1786007691360835 us; error 45 us; skew 500 ppm
I20260806 09:14:51.361217 26710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:51 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007691, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:51.364480 26710 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:14:51.365865 26710 webserver.cc:460] Webserver started at http://127.3.69.2:46471/ using document root <none> and password file <none>
I20260806 09:14:51.366499 26710 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:51.366623 26710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:51.373436 26710 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:14:51.376150 26730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:51.377594 26710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:14:51.377739 26710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "d543852c35d1474691475d9bf606d5e0"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "cfe5f06b24bb48958ddf167cd6ee86f5"
server_key_iv: "6344a03a1b964e6927fec2afed7cb2ce"
server_key_version: "encryptionkey@0"
I20260806 09:14:51.378677 26710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:51.392217 26710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:51.396510 26710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:51.396737 26710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:51.397653 26710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:51.398965 26710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:51.399053 26710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:51.399194 26710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:51.399257 26710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:51.434679 26710 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:41509
I20260806 09:14:51.434767 26894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:41509 every 8 connection(s)
I20260806 09:14:51.436187 26710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:14:51.446069 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 26710
I20260806 09:14:51.446774 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:46699
--local_ip_for_outbound_sockets=127.3.69.3
--tserver_master_addrs=127.3.69.62:45345
--webserver_port=37485
--webserver_interface=127.3.69.3
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:14:51 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007691, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:51.456336 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.438862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:38123 (local address 127.3.69.62:45345)
0806 09:14:51.439121 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:51.439129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:51.440291 (+ 1162us) server_negotiation.cc:407] Connection header received
0806 09:14:51.442155 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:51.442160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:51.442244 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:51.442337 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:51.443927 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.444738 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.446347 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.446758 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.446887 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:51.450459 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:51.450464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:51.450467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:51.450515 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:51.453147 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:51.453907 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:51.453912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:51.453914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:51.454041 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:51.454487 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:51.454491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:51.454493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:51.454779 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:51.454938 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:51.455714 (+ 776us) server_negotiation.cc:299] Negotiation successful
0806 09:14:51.455971 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:14:51.457356 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.439398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:38123)
0806 09:14:51.440121 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:51.440166 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:51.441875 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:51.442581 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:51.442589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:51.443186 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:51.443766 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:51.443783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.444901 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.444905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:51.446180 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:51.446199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.447000 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.447008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:51.447288 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:51.447988 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:51.448013 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:51.450213 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:51.453297 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:51.453306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:51.453323 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:51.453771 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:51.454151 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:51.454156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:51.454160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:51.454356 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:51.454916 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:51.454924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:51.455343 (+ 419us) client_negotiation.cc:769] Sending connection context
0806 09:14:51.455620 (+ 277us) client_negotiation.cc:240] Negotiation successful
0806 09:14:51.456190 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":286,"threads_started":1}
I20260806 09:14:51.458998 26897 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45345
I20260806 09:14:51.459388 26897 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:51.460148 26897 heartbeater.cc:511] Master 127.3.69.62:45345 requested a full tablet report, sending...
I20260806 09:14:51.461795 26419 ts_manager.cc:194] Registered new tserver with Master: d543852c35d1474691475d9bf606d5e0 (127.3.69.2:41509)
I20260806 09:14:51.462622 26419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:38123
W20260806 09:14:51.796406 26900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:14:51.796734 26900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:14:51.796905 26900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:14:51.804562 26900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:14:51.804703 26900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:14:51.804886 26900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:14:51.814465 26900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:46699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=37485
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:14:51.815865 26900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:14:51.817198 26900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:14:51.830693 26907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:51.830650 26908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:14:51.830592 26911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:14:51.831727 26900 server_base.cc:1034] running on GCE node
I20260806 09:14:51.832348 26900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:14:51.833274 26900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:14:51.834508 26900 hybrid_clock.cc:648] HybridClock initialized: now 1786007691834459 us; error 55 us; skew 500 ppm
I20260806 09:14:51.834848 26900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:14:51 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007691, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:14:51.838106 26900 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:14:51.839807 26900 webserver.cc:460] Webserver started at http://127.3.69.3:37485/ using document root <none> and password file <none>
I20260806 09:14:51.840517 26900 fs_manager.cc:359] Metadata directory not provided
I20260806 09:14:51.840687 26900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:14:51.846127 26900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 09:14:51.848835 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:51.850178 26900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 09:14:51.850363 26900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "75153d0835724ccbae2d4e66bc1e049b"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "0b794f8db22c0d8fa95c031578df7071"
server_key_iv: "a8bb0a92262e3af35e6a76cc5b17c880"
server_key_version: "encryptionkey@0"
I20260806 09:14:51.851434 26900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:14:51.872028 26900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:14:51.876525 26900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:14:51.876834 26900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:14:51.877924 26900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:14:51.879364 26900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:14:51.879498 26900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:51.879640 26900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:14:51.879706 26900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:14:51.913990 26900 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:46699
I20260806 09:14:51.914065 27083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:46699 every 8 connection(s)
I20260806 09:14:51.915477 26900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:14:51.920521 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 26900
Aug 06 09:14:51 dist-test-slave-w4v5 krb5kdc[25534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007691, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:14:51.936831 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.917943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:58845 (local address 127.3.69.62:45345)
0806 09:14:51.918162 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:51.918169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:51.919274 (+ 1105us) server_negotiation.cc:407] Connection header received
0806 09:14:51.921084 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:51.921096 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:51.921166 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:51.921255 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:51.923105 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.924322 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.925845 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.926458 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.926582 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:51.930296 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:51.930304 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:51.930309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:51.930353 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:51.933388 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:51.934276 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:51.934280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:51.934282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:51.934398 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:51.934833 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:51.934839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:51.934841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:51.935228 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:51.935359 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:51.936201 (+ 842us) server_negotiation.cc:299] Negotiation successful
0806 09:14:51.936421 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:14:51.937311 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.921542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.3.69.62:45345)
0806 09:14:51.922073 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:51.922081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:51.922403 (+ 322us) server_negotiation.cc:407] Connection header received
0806 09:14:51.922603 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:51.922608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:51.922670 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:51.922781 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:51.924785 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.925494 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.927249 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.927648 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.927783 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:51.931058 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:51.931066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:51.931097 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:51.931150 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:51.934376 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:51.935171 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:51.935174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:51.935177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:51.935348 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:51.935739 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:51.935744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:51.935747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:51.936106 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:51.936367 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:51.936697 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 09:14:51.936902 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":301,"threads_started":1}
I20260806 09:14:51.938192 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.918364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:58845)
0806 09:14:51.919059 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:51.919166 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:51.920821 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:51.921548 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:51.921560 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:51.922378 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:14:51.922923 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:51.922939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.924477 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.924480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:51.925711 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:51.925724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.926665 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.926671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:51.926859 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:51.927545 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:14:51.927572 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:14:51.930095 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:14:51.933523 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:51.933532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:51.933548 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:51.934164 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:51.934535 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:14:51.934539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:14:51.934541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:14:51.934718 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:14:51.935349 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:14:51.935357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:14:51.935772 (+ 415us) client_negotiation.cc:769] Sending connection context
0806 09:14:51.936117 (+ 345us) client_negotiation.cc:240] Negotiation successful
0806 09:14:51.936636 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":269,"threads_started":1}
I20260806 09:14:51.939690 27084 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45345
I20260806 09:14:51.940008 27084 heartbeater.cc:464] Registering TS with master...
I20260806 09:14:51.940667 27084 heartbeater.cc:511] Master 127.3.69.62:45345 requested a full tablet report, sending...
I20260806 09:14:51.941974 26419 ts_manager.cc:194] Registered new tserver with Master: 75153d0835724ccbae2d4e66bc1e049b (127.3.69.3:46699)
I20260806 09:14:51.942749 26419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:58845
I20260806 09:14:51.952198 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:14:51.958420 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.953073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.3.69.62:45345)
0806 09:14:51.953301 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:51.953310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:51.953676 (+ 366us) server_negotiation.cc:407] Connection header received
0806 09:14:51.953869 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:51.953873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:51.953992 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:51.954113 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:14:51.954504 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.955331 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.956695 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.957099 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.957194 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:51.957638 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:14:51.957915 (+ 277us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 09:14:51.958192 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59934: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 09:14:51.958546 27089 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59934: token expired
W20260806 09:14:51.959177 26305 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.69.62:45345)
I20260806 09:14:51.972393 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.959617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.3.69.62:45345)
0806 09:14:51.959758 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:51.959762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:51.959822 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:14:51.959937 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:51.959941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:51.959993 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:51.960068 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:51.961372 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.962101 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.963469 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:51.963863 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:51.963952 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:51.966627 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:51.966634 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:51.966638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:51.966675 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:51.969952 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:51.970655 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:51.970659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:51.970662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:51.970788 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:51.971145 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:51.971161 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:51.971167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:51.971478 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:51.971656 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:51.971990 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 09:14:51.972147 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 09:14:51.975558 26419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:14:51.982957 26419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:14:52.006839 26702 heartbeater.cc:503] Master 127.3.69.62:45345 was elected leader, sending a full tablet report...
I20260806 09:14:52.020563 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.994564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.3.69.3:46699)
0806 09:14:52.010002 (+ 15438us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.010016 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.010060 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:14:52.010182 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.010190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.010520 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.010770 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:52.013649 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.015441 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.017932 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.019147 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.019750 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.019999 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.020074 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.020237 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15188,"thread_start_us":16507,"threads_started":1}
I20260806 09:14:52.020704 27099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.994515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:46699 (local address 127.0.0.1:38074)
0806 09:14:52.006657 (+ 12142us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:52.006705 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:52.006890 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:52.012960 (+ 6070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:52.012964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:52.012990 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:52.013427 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.013439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.015599 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.015604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.017773 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.017786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.019449 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.019456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.019780 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.019839 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 09:14:52.019970 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 09:14:52.020171 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11761,"mutex_wait_us":106,"thread_start_us":12564,"threads_started":1}
I20260806 09:14:52.021016 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.994746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46843 (local address 127.0.0.1:45824)
0806 09:14:52.008047 (+ 13301us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:52.008071 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:52.008227 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:52.012624 (+ 4397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:52.012631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:52.012663 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:52.013206 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.013220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.015598 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.015604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.018077 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.018093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.020090 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.020097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.020266 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.020298 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:14:52.020365 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 09:14:52.020514 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13139,"mutex_wait_us":47,"thread_start_us":13543,"threads_started":1}
I20260806 09:14:52.022379 27106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:51.995675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.3.69.1:46843)
0806 09:14:52.012029 (+ 16354us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.012035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.012056 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:14:52.012133 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.012137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.012385 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.013461 (+ 1076us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:52.013728 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.015429 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.018265 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.019838 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.021558 (+ 1720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.021785 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.021866 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.022031 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16192,"thread_start_us":16494,"threads_started":1}
I20260806 09:14:52.024030 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.008774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:41509 (local address 127.0.0.1:43258)
0806 09:14:52.011717 (+ 2943us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:52.011738 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:52.011855 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:52.016249 (+ 4394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:52.016254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:52.016280 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:52.016756 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.016769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.018731 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.018738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.021034 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.021052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.023208 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.023214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.023376 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.023399 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 09:14:52.023510 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 09:14:52.023631 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2801,"mutex_wait_us":21,"thread_start_us":2671,"threads_started":1}
I20260806 09:14:52.024063 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.009105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.3.69.2:41509)
0806 09:14:52.015617 (+ 6512us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.015623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.015669 (+ 46us) server_negotiation.cc:407] Connection header received
0806 09:14:52.015766 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.015774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.016020 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.016221 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:52.016971 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.018563 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.021227 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.022700 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.022830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.023011 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.023561 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.023729 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6361,"thread_start_us":6219,"threads_started":1}
I20260806 09:14:52.024749 26991 tablet_service.cc:1467] Processing CreateTablet for tablet 0a573163d1b7477c8b55040ade42dce4 (DEFAULT_TABLE table=test-table [id=c23a037949ed404fa4028f3d6df4a75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:52.026342 26991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a573163d1b7477c8b55040ade42dce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:52.025872 26604 tablet_service.cc:1467] Processing CreateTablet for tablet 0a573163d1b7477c8b55040ade42dce4 (DEFAULT_TABLE table=test-table [id=c23a037949ed404fa4028f3d6df4a75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:52.027330 26604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a573163d1b7477c8b55040ade42dce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:52.027372 26799 tablet_service.cc:1467] Processing CreateTablet for tablet 0a573163d1b7477c8b55040ade42dce4 (DEFAULT_TABLE table=test-table [id=c23a037949ed404fa4028f3d6df4a75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:14:52.028604 26799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a573163d1b7477c8b55040ade42dce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:14:52.040279 27109 tablet_bootstrap.cc:492] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: Bootstrap starting.
I20260806 09:14:52.042351 27112 tablet_bootstrap.cc:492] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Bootstrap starting.
I20260806 09:14:52.043182 27111 tablet_bootstrap.cc:492] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: Bootstrap starting.
I20260806 09:14:52.044626 27109 tablet_bootstrap.cc:654] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:52.046675 27112 tablet_bootstrap.cc:654] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:52.046722 27109 log.cc:824] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:52.046911 27111 tablet_bootstrap.cc:654] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: Neither blocks nor log segments found. Creating new log.
I20260806 09:14:52.048019 27112 log.cc:824] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:52.048063 27111 log.cc:824] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: Log is configured to *not* fsync() on all Append() calls
I20260806 09:14:52.049844 27109 tablet_bootstrap.cc:492] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: No bootstrap required, opened a new log
I20260806 09:14:52.050117 27109 ts_tablet_manager.cc:1397] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260806 09:14:52.050648 27112 tablet_bootstrap.cc:492] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: No bootstrap required, opened a new log
I20260806 09:14:52.050889 27112 ts_tablet_manager.cc:1397] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260806 09:14:52.053066 27111 tablet_bootstrap.cc:492] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: No bootstrap required, opened a new log
I20260806 09:14:52.053334 27111 ts_tablet_manager.cc:1397] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 09:14:52.058246 27112 raft_consensus.cc:348] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.058782 27112 raft_consensus.cc:374] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:52.058938 27112 raft_consensus.cc:729] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75153d0835724ccbae2d4e66bc1e049b, State: Initialized, Role: FOLLOWER
I20260806 09:14:52.059597 27112 consensus_queue.cc:260] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.061604 27109 raft_consensus.cc:348] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.062077 27109 raft_consensus.cc:374] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:52.062254 27109 raft_consensus.cc:729] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d543852c35d1474691475d9bf606d5e0, State: Initialized, Role: FOLLOWER
I20260806 09:14:52.062755 27084 heartbeater.cc:503] Master 127.3.69.62:45345 was elected leader, sending a full tablet report...
I20260806 09:14:52.063100 27109 consensus_queue.cc:260] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.063508 27112 ts_tablet_manager.cc:1428] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260806 09:14:52.064036 27111 raft_consensus.cc:348] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.064522 26897 heartbeater.cc:503] Master 127.3.69.62:45345 was elected leader, sending a full tablet report...
I20260806 09:14:52.064678 27111 raft_consensus.cc:374] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:14:52.064826 27111 raft_consensus.cc:729] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613ee6916e124eab9982c2633965dd09, State: Initialized, Role: FOLLOWER
I20260806 09:14:52.065426 27111 consensus_queue.cc:260] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.067768 27109 ts_tablet_manager.cc:1428] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: Time spent starting tablet: real 0.018s user 0.015s sys 0.001s
I20260806 09:14:52.069527 27111 ts_tablet_manager.cc:1428] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260806 09:14:52.107894 27117 raft_consensus.cc:482] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:14:52.108222 27117 raft_consensus.cc:504] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.109946 27117 leader_election.cc:288] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 613ee6916e124eab9982c2633965dd09 (127.3.69.1:46843), d543852c35d1474691475d9bf606d5e0 (127.3.69.2:41509)
I20260806 09:14:52.115688 27118 raft_consensus.cc:482] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:14:52.115804 27087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.110333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46843 (local address 127.3.69.3:35269)
0806 09:14:52.110604 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:52.110627 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:52.110799 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:52.111439 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:52.111444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:52.111469 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:52.111670 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.111677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.112659 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.112664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.113953 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.113969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.115049 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.115055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.115275 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.115300 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:14:52.115399 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:14:52.115528 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 09:14:52.115823 27106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.110402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:35269 (local address 127.3.69.1:46843)
0806 09:14:52.110621 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.110628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.110715 (+ 87us) server_negotiation.cc:407] Connection header received
0806 09:14:52.111003 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.111011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.111146 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.111290 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:52.111811 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.112532 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.114109 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.114896 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.115040 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.115171 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.115442 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.115566 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:14:52.116039 27118 raft_consensus.cc:504] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.116951 26635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a573163d1b7477c8b55040ade42dce4" candidate_uuid: "75153d0835724ccbae2d4e66bc1e049b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613ee6916e124eab9982c2633965dd09" is_pre_election: true
I20260806 09:14:52.117416 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.111062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:38813 (local address 127.3.69.2:41509)
0806 09:14:52.111366 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.111373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.111681 (+ 308us) server_negotiation.cc:407] Connection header received
0806 09:14:52.111869 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.111874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.111940 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.112058 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:52.112585 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.113285 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.115056 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.116366 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.116505 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.116607 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.117031 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.117172 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 09:14:52.117445 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.110762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:41509 (local address 127.3.69.3:38813)
0806 09:14:52.111568 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:52.111589 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:52.111740 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:52.112069 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:52.112073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:52.112097 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:52.112425 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.112434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.113439 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.113444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.114890 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.114902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.116627 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.116635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.116827 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.116864 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:14:52.116970 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 09:14:52.117106 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":441,"threads_started":1}
I20260806 09:14:52.117528 26635 raft_consensus.cc:2454] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75153d0835724ccbae2d4e66bc1e049b in term 0.
I20260806 09:14:52.118132 27118 leader_election.cc:288] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75153d0835724ccbae2d4e66bc1e049b (127.3.69.3:46699), 613ee6916e124eab9982c2633965dd09 (127.3.69.1:46843)
I20260806 09:14:52.118360 26830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a573163d1b7477c8b55040ade42dce4" candidate_uuid: "75153d0835724ccbae2d4e66bc1e049b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d543852c35d1474691475d9bf606d5e0" is_pre_election: true
I20260806 09:14:52.118451 26928 leader_election.cc:302] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613ee6916e124eab9982c2633965dd09, 75153d0835724ccbae2d4e66bc1e049b; no voters:
I20260806 09:14:52.118825 26830 raft_consensus.cc:2454] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75153d0835724ccbae2d4e66bc1e049b in term 0.
I20260806 09:14:52.119376 27117 raft_consensus.cc:2781] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:14:52.119553 27117 raft_consensus.cc:482] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:14:52.119659 27117 raft_consensus.cc:3037] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:52.123535 27117 raft_consensus.cc:504] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.124256 27117 leader_election.cc:288] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [CANDIDATE]: Term 1 election: Requested vote from peers 613ee6916e124eab9982c2633965dd09 (127.3.69.1:46843), d543852c35d1474691475d9bf606d5e0 (127.3.69.2:41509)
I20260806 09:14:52.124771 26830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a573163d1b7477c8b55040ade42dce4" candidate_uuid: "75153d0835724ccbae2d4e66bc1e049b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d543852c35d1474691475d9bf606d5e0"
I20260806 09:14:52.124996 26830 raft_consensus.cc:3037] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:52.125661 26635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a573163d1b7477c8b55040ade42dce4" candidate_uuid: "75153d0835724ccbae2d4e66bc1e049b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613ee6916e124eab9982c2633965dd09"
I20260806 09:14:52.125885 26635 raft_consensus.cc:3037] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:14:52.127993 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.118275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:46699 (local address 127.3.69.2:60239)
0806 09:14:52.120734 (+ 2459us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:52.120753 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:52.120926 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:52.121340 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:52.121344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:52.121365 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:52.121666 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.121673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.123101 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.123106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.125302 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.125322 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.127146 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.127153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.127347 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.127385 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:14:52.127482 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:14:52.127629 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2299,"thread_start_us":3669,"threads_started":1}
I20260806 09:14:52.128598 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.118982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:60239 (local address 127.3.69.3:46699)
0806 09:14:52.119247 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.119256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.120824 (+ 1568us) server_negotiation.cc:407] Connection header received
0806 09:14:52.121090 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.121093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.121146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.121195 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:52.121805 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.122919 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.125498 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.126701 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.126828 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.126948 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.128212 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.128347 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 09:14:52.128576 26830 raft_consensus.cc:2454] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75153d0835724ccbae2d4e66bc1e049b in term 1.
I20260806 09:14:52.128892 26928 leader_election.cc:302] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75153d0835724ccbae2d4e66bc1e049b, d543852c35d1474691475d9bf606d5e0; no voters:
I20260806 09:14:52.129244 26635 raft_consensus.cc:2454] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75153d0835724ccbae2d4e66bc1e049b in term 1.
I20260806 09:14:52.129338 27117 raft_consensus.cc:2781] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:14:52.129938 27117 raft_consensus.cc:686] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 1 LEADER]: Becoming Leader. State: Replica: 75153d0835724ccbae2d4e66bc1e049b, State: Running, Role: LEADER
I20260806 09:14:52.129940 27024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a573163d1b7477c8b55040ade42dce4" candidate_uuid: "d543852c35d1474691475d9bf606d5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75153d0835724ccbae2d4e66bc1e049b" is_pre_election: true
I20260806 09:14:52.130499 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.119517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46843 (local address 127.3.69.2:45575)
0806 09:14:52.122243 (+ 2726us) negotiation.cc:107] Waiting for socket to connect
0806 09:14:52.122261 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:14:52.122383 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:14:52.122701 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:14:52.122705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:14:52.122730 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:14:52.123153 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.123164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.124583 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.124591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.126793 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:14:52.126808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.128949 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.128955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:14:52.129156 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.129187 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 09:14:52.129989 (+ 802us) client_negotiation.cc:240] Negotiation successful
0806 09:14:52.130123 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2606,"mutex_wait_us":24,"thread_start_us":875,"threads_started":1}
I20260806 09:14:52.130715 27117 consensus_queue.cc:237] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } }
I20260806 09:14:52.131270 27106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.119954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:45575 (local address 127.3.69.1:46843)
0806 09:14:52.120133 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.120139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.122354 (+ 2215us) server_negotiation.cc:407] Connection header received
0806 09:14:52.122507 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.122512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.122579 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.122672 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:14:52.123317 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.124234 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.127010 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.128430 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.128523 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.128629 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.129419 (+ 790us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.130959 (+ 1540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:14:52.131745 26635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a573163d1b7477c8b55040ade42dce4" candidate_uuid: "d543852c35d1474691475d9bf606d5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613ee6916e124eab9982c2633965dd09" is_pre_election: true
I20260806 09:14:52.132027 26635 raft_consensus.cc:2379] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d543852c35d1474691475d9bf606d5e0 in current term 1: Already voted for candidate 75153d0835724ccbae2d4e66bc1e049b in this term.
I20260806 09:14:52.132637 26738 leader_election.cc:302] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d543852c35d1474691475d9bf606d5e0; no voters: 613ee6916e124eab9982c2633965dd09, 75153d0835724ccbae2d4e66bc1e049b
I20260806 09:14:52.133047 27118 raft_consensus.cc:2726] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 09:14:52.135515 26419 catalog_manager.cc:5495] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b reported cstate change: term changed from 0 to 1, leader changed from <none> to 75153d0835724ccbae2d4e66bc1e049b (127.3.69.3). New cstate: current_term: 1 leader_uuid: "75153d0835724ccbae2d4e66bc1e049b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75153d0835724ccbae2d4e66bc1e049b" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 46699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 } health_report { overall_health: UNKNOWN } } }
I20260806 09:14:52.160955 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.156193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.3.69.3:46699)
0806 09:14:52.156382 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.156386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.156403 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:14:52.156483 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.156487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.156585 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.156708 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:14:52.157017 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.157752 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.159044 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.159446 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.159537 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.159998 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:14:52.160473 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:14:52.160581 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.160672 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.160736 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 09:14:52.166988 27085 tablet.cc:2369] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:14:52.169112 26635 raft_consensus.cc:1261] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 1 FOLLOWER]: Refusing update from remote peer 75153d0835724ccbae2d4e66bc1e049b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:14:52.169178 26830 raft_consensus.cc:1261] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Refusing update from remote peer 75153d0835724ccbae2d4e66bc1e049b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:14:52.169991 27117 consensus_queue.cc:1035] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [LEADER]: Connected to new peer: Peer: permanent_uuid: "613ee6916e124eab9982c2633965dd09" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:14:52.170300 27126 consensus_queue.cc:1035] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d543852c35d1474691475d9bf606d5e0" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:14:52.197517 26416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:14:52.197839 26416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:14:52.201511 26416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Sending DeleteTablet for 3 replicas of tablet 0a573163d1b7477c8b55040ade42dce4
I20260806 09:14:52.202626 26991 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a573163d1b7477c8b55040ade42dce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:52 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:38074
I20260806 09:14:52.202760 26604 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a573163d1b7477c8b55040ade42dce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:52 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:45824
I20260806 09:14:52.203459 26799 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a573163d1b7477c8b55040ade42dce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:14:52 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:43258
I20260806 09:14:52.204561 27148 tablet_replica.cc:330] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: stopping tablet replica
I20260806 09:14:52.205163 27148 raft_consensus.cc:2229] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:14:52.205271 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 26478
I20260806 09:14:52.205837 27148 raft_consensus.cc:2258] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:52.208503 27148 ts_tablet_manager.cc:1905] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:52.213517 27147 tablet_replica.cc:330] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: stopping tablet replica
I20260806 09:14:52.214109 27147 raft_consensus.cc:2229] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:14:52.214740 27147 raft_consensus.cc:2258] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:52.215950 27148 ts_tablet_manager.cc:1918] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:52.216202 27148 log.cc:1196] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/0a573163d1b7477c8b55040ade42dce4
I20260806 09:14:52.216938 27148 ts_tablet_manager.cc:1939] T 0a573163d1b7477c8b55040ade42dce4 P d543852c35d1474691475d9bf606d5e0: Deleting consensus metadata
I20260806 09:14:52.217257 27147 ts_tablet_manager.cc:1905] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:52.219264 26399 catalog_manager.cc:4841] TS d543852c35d1474691475d9bf606d5e0 (127.3.69.2:41509): tablet 0a573163d1b7477c8b55040ade42dce4 (table test-table [id=c23a037949ed404fa4028f3d6df4a75c]) successfully deleted
I20260806 09:14:52.220197 27146 tablet_replica.cc:330] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: stopping tablet replica
I20260806 09:14:52.220993 27146 raft_consensus.cc:2229] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:14:52.222059 27146 raft_consensus.cc:2258] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:14:52.224341 27146 ts_tablet_manager.cc:1905] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:14:52.225011 27147 ts_tablet_manager.cc:1918] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:52.225261 27147 log.cc:1196] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/0a573163d1b7477c8b55040ade42dce4
I20260806 09:14:52.225905 27147 ts_tablet_manager.cc:1939] T 0a573163d1b7477c8b55040ade42dce4 P 613ee6916e124eab9982c2633965dd09: Deleting consensus metadata
I20260806 09:14:52.227898 26399 catalog_manager.cc:4841] TS 613ee6916e124eab9982c2633965dd09 (127.3.69.1:46843): tablet 0a573163d1b7477c8b55040ade42dce4 (table test-table [id=c23a037949ed404fa4028f3d6df4a75c]) successfully deleted
I20260806 09:14:52.228721 27106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.207195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.3.69.1:46843)
0806 09:14:52.207418 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.207426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.207450 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:14:52.207523 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.207531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.207642 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.207763 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:52.210328 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.211895 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.214280 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.214934 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.215062 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.219742 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:52.219760 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:52.219783 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:52.219831 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:52.224342 (+ 4511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:52.225324 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:52.225333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:52.225337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:52.225504 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:52.226033 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:52.226039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:52.226043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:52.226517 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:52.227936 (+ 1419us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.228092 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.228327 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:14:52.229452 26695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:52.233603 27146 ts_tablet_manager.cc:1918] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:14:52.233807 27146 log.cc:1196] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/0a573163d1b7477c8b55040ade42dce4
I20260806 09:14:52.234360 27146 ts_tablet_manager.cc:1939] T 0a573163d1b7477c8b55040ade42dce4 P 75153d0835724ccbae2d4e66bc1e049b: Deleting consensus metadata
I20260806 09:14:52.236023 26398 catalog_manager.cc:4841] TS 75153d0835724ccbae2d4e66bc1e049b (127.3.69.3:46699): tablet 0a573163d1b7477c8b55040ade42dce4 (table test-table [id=c23a037949ed404fa4028f3d6df4a75c]) successfully deleted
I20260806 09:14:52.339986 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 26478
I20260806 09:14:52.371675 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 26710
I20260806 09:14:52.388661 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.372496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.3.69.2:41509)
0806 09:14:52.372700 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.372709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.373046 (+ 337us) server_negotiation.cc:407] Connection header received
0806 09:14:52.373256 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.373262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.373423 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.373532 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:52.375887 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.376883 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.378850 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.379389 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.379527 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.382442 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:52.382465 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:52.382483 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:52.382517 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:52.385554 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:52.386327 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:52.386337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:52.386341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:52.386478 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:52.386873 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:52.386879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:52.386882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:52.387359 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:52.387665 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.388054 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.388310 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:14:52.389369 26889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:52.482889 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 26710
I20260806 09:14:52.510730 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 26900
I20260806 09:14:52.527810 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:14:52.511559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.3.69.3:46699)
0806 09:14:52.511776 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:14:52.511787 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:14:52.511813 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:14:52.511943 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:14:52.511948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:14:52.512082 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:14:52.512213 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:14:52.513651 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.514941 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.516485 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:14:52.517228 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:14:52.517368 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:14:52.520200 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:14:52.520217 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:14:52.520241 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:14:52.520292 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:14:52.524588 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:52.525322 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:52.525332 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:52.525336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:52.525500 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:14:52.525890 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:14:52.525896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:14:52.525899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:14:52.526435 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:14:52.526818 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0806 09:14:52.527160 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 09:14:52.527395 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:14:52.528625 27079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:52.659405 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 26900
I20260806 09:14:52.688995 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 26369
I20260806 09:14:52.690203 26464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:14:52.810681 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 26369
I20260806 09:15:12.842888 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45345
--webserver_interface=127.3.69.62
--webserver_port=42821
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:45345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:13.194274 27156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:13.194581 27156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:13.194672 27156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:13.202207 27156 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:15:13.202309 27156 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:15:13.202348 27156 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:15:13.202401 27156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:13.202456 27156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:13.202524 27156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:13.202571 27156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:13.202620 27156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:13.213279 27156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:45345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=42821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:13.214512 27156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:13.215853 27156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:13.228034 27161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:13.227988 27162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:13.228075 27166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:13.229255 27156 server_base.cc:1034] running on GCE node
I20260806 09:15:13.230228 27156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:13.231742 27156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:13.233032 27156 hybrid_clock.cc:648] HybridClock initialized: now 1786007713232941 us; error 36 us; skew 500 ppm
I20260806 09:15:13.233371 27156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:13 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007713, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:13.239791 27156 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:15:13.241353 27156 webserver.cc:460] Webserver started at http://127.3.69.62:42821/ using document root <none> and password file <none>
I20260806 09:15:13.242069 27156 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:13.242188 27156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:13.247625 27156 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:15:13.250564 27176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:13.252090 27156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:15:13.252277 27156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "0b54eabcfc534e3598ac36fa9026c15b"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "40b943e94ec36e1d2079f361ecad61f5"
server_key_iv: "2375387cebe5272084fd7b9a27c732a0"
server_key_version: "encryptionkey@0"
I20260806 09:15:13.253310 27156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:13.287235 27156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:13.291594 27156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:13.291903 27156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:13.311467 27156 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:45345
I20260806 09:15:13.311519 27258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:45345 every 8 connection(s)
I20260806 09:15:13.312995 27156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:13.315268 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 27156
I20260806 09:15:13.316129 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:46843
--local_ip_for_outbound_sockets=127.3.69.1
--tserver_master_addrs=127.3.69.62:45345
--webserver_port=45211
--webserver_interface=127.3.69.1
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:15:13.320423 27259 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:15:13.323814 27259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Bootstrap starting.
I20260806 09:15:13.345384 27259 log.cc:824] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:13.362118 27259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:15:13.362689 27259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Bootstrap complete.
I20260806 09:15:13.372166 27259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:15:13.373044 27259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b54eabcfc534e3598ac36fa9026c15b, State: Initialized, Role: FOLLOWER
I20260806 09:15:13.373709 27259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:15:13.373946 27259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:13.374063 27259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:13.374277 27259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 2 FOLLOWER]: Advancing to term 3
I20260806 09:15:13.378132 27259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:15:13.378698 27259 leader_election.cc:302] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b54eabcfc534e3598ac36fa9026c15b; no voters:
I20260806 09:15:13.379426 27259 leader_election.cc:288] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 09:15:13.379567 27265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 3 FOLLOWER]: Leader election won for term 3
I20260806 09:15:13.380784 27265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [term 3 LEADER]: Becoming Leader. State: Replica: 0b54eabcfc534e3598ac36fa9026c15b, State: Running, Role: LEADER
I20260806 09:15:13.381430 27265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } }
I20260806 09:15:13.381709 27259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:13.383891 27274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b54eabcfc534e3598ac36fa9026c15b. Latest consensus state: current_term: 3 leader_uuid: "0b54eabcfc534e3598ac36fa9026c15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } } }
I20260806 09:15:13.384285 27274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:13.384847 27270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0b54eabcfc534e3598ac36fa9026c15b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b54eabcfc534e3598ac36fa9026c15b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45345 } } }
I20260806 09:15:13.385130 27270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:13.387655 27279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:13.396450 27279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:13.397956 27279 catalog_manager.cc:1259] Loaded cluster ID: beaa5d7ade304422ab2113f2fe541288
I20260806 09:15:13.398123 27279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:13.402598 27279 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:13.404361 27279 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Loaded TSK: 0
I20260806 09:15:13.404588 27279 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b54eabcfc534e3598ac36fa9026c15b: Loaded TSK: 1
F20260806 09:15:13.405794 27279 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f400a9119ad google::LogMessage::Fail() at ??:0
@ 0x7f400a91488a google::LogMessage::SendToLog() at ??:0
@ 0x7f400a9114de google::LogMessage::Flush() at ??:0
@ 0x7f400a912a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f400e6757d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f400e66473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f40203401c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f402033f783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f40203cb2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f40203cb0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f40204155e9 std::function<>::operator()() at ??:0
@ 0x7f4020343cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4020341c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f40203c9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f40203c9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f400d55d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f400d56091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f400d560799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f400d53beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f40108156db start_thread at ??:0
@ 0x7f40097ab61f clone at ??:0
W20260806 09:15:13.718420 27261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:13.718799 27261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:13.718979 27261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:13.726358 27261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:13.726523 27261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:13.726706 27261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:15:13.736974 27261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:46843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=45211
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:13.738313 27261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:13.739771 27261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:13.753105 27335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:13.753082 27339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:13.753151 27333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:13.753991 27261 server_base.cc:1034] running on GCE node
I20260806 09:15:13.754582 27261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:13.755586 27261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:13.756781 27261 hybrid_clock.cc:648] HybridClock initialized: now 1786007713756703 us; error 54 us; skew 500 ppm
I20260806 09:15:13.757117 27261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:13 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007713, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:13.760463 27261 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:15:13.761876 27261 webserver.cc:460] Webserver started at http://127.3.69.1:45211/ using document root <none> and password file <none>
I20260806 09:15:13.762524 27261 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:13.762629 27261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:13.768025 27261 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:15:13.770792 27346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:13.772200 27261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:15:13.772384 27261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "613ee6916e124eab9982c2633965dd09"
format_stamp: "Formatted at 2026-08-06 09:14:27 on dist-test-slave-w4v5"
server_key: "c2cb1645ac098f3a13f521d00faf8b7b"
server_key_iv: "01c5bdd4c38e43dbe44221ee57772ecb"
server_key_version: "encryptionkey@0"
I20260806 09:15:13.773397 27261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:13.784382 27261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:13.788936 27261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:13.789235 27261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:13.790269 27261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:13.791764 27261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:13.791854 27261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:13.792012 27261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:13.792121 27261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:13.827869 27261 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:46843
I20260806 09:15:13.827943 27518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:46843 every 8 connection(s)
I20260806 09:15:13.829505 27261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:15:13.834687 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:13.832807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:37059)
0806 09:15:13.833616 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:13.833819 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":352,"threads_started":1}
W20260806 09:15:13.835331 27519 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:45345 (0 consecutive failures): Network error: Failed to ping master at 127.3.69.62:45345: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
I20260806 09:15:13.836216 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:13.835887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:39933)
0806 09:15:13.836029 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:13.836096 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 09:15:13.837113 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:13.836741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:60779)
0806 09:15:13.836920 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:13.837001 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 09:15:13.837440 27519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:15:13.838717 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 27261
I20260806 09:15:13.839519 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:41509
--local_ip_for_outbound_sockets=127.3.69.2
--tserver_master_addrs=127.3.69.62:45345
--webserver_port=46471
--webserver_interface=127.3.69.2
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:14.193200 27522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:14.193481 27522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:14.193588 27522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:14.200896 27522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:14.201009 27522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:14.201182 27522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:15:14.210888 27522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:41509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=46471
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:14.212188 27522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:14.213560 27522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:14.227216 27530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:14.227128 27533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:14.227169 27531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:14.228159 27522 server_base.cc:1034] running on GCE node
I20260806 09:15:14.228849 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:14.229914 27522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:14.231186 27522 hybrid_clock.cc:648] HybridClock initialized: now 1786007714231127 us; error 66 us; skew 500 ppm
I20260806 09:15:14.231545 27522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:14 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007714, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:14.234956 27522 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:15:14.237003 27522 webserver.cc:460] Webserver started at http://127.3.69.2:46471/ using document root <none> and password file <none>
I20260806 09:15:14.237703 27522 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:14.237810 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:14.243458 27522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:15:14.246239 27540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:14.247680 27522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:15:14.247859 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "d543852c35d1474691475d9bf606d5e0"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "cfe5f06b24bb48958ddf167cd6ee86f5"
server_key_iv: "6344a03a1b964e6927fec2afed7cb2ce"
server_key_version: "encryptionkey@0"
I20260806 09:15:14.248839 27522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:14.278576 27522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:14.283195 27522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:14.283486 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:14.284538 27522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:14.285925 27522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:14.286015 27522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:14.286171 27522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:14.286264 27522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:14.321718 27522 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:41509
I20260806 09:15:14.321810 27710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:41509 every 8 connection(s)
I20260806 09:15:14.323330 27522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:15:14.328217 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:14.326325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:58431)
0806 09:15:14.327117 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:14.327356 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":298,"threads_started":1}
W20260806 09:15:14.328934 27711 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:45345 (0 consecutive failures): Network error: Failed to ping master at 127.3.69.62:45345: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
I20260806 09:15:14.329841 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:14.329499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:54847)
0806 09:15:14.329670 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:14.329735 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 09:15:14.330754 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:14.330439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:50357)
0806 09:15:14.330563 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:14.330632 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 09:15:14.331012 27711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:15:14.333669 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 27522
I20260806 09:15:14.334295 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:46699
--local_ip_for_outbound_sockets=127.3.69.3
--tserver_master_addrs=127.3.69.62:45345
--webserver_port=37485
--webserver_interface=127.3.69.3
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:14.689872 27715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:14.690198 27715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:14.690366 27715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:14.697768 27715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:14.697938 27715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:14.698134 27715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:15:14.708329 27715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:46699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=37485
--tserver_master_addrs=127.3.69.62:45345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:14.709689 27715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:14.711062 27715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:14.724589 27722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:14.724583 27726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:14.724622 27720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:14.725557 27715 server_base.cc:1034] running on GCE node
I20260806 09:15:14.726276 27715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:14.727314 27715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:14.728581 27715 hybrid_clock.cc:648] HybridClock initialized: now 1786007714728494 us; error 64 us; skew 500 ppm
I20260806 09:15:14.728914 27715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:14 dist-test-slave-w4v5 krb5kdc[25534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007714, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:14.732344 27715 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:15:14.733930 27715 webserver.cc:460] Webserver started at http://127.3.69.3:37485/ using document root <none> and password file <none>
I20260806 09:15:14.734637 27715 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:14.734793 27715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:14.740293 27715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:15:14.743213 27734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:14.744560 27715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:15:14.744751 27715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "75153d0835724ccbae2d4e66bc1e049b"
format_stamp: "Formatted at 2026-08-06 09:14:28 on dist-test-slave-w4v5"
server_key: "0b794f8db22c0d8fa95c031578df7071"
server_key_iv: "a8bb0a92262e3af35e6a76cc5b17c880"
server_key_version: "encryptionkey@0"
I20260806 09:15:14.745775 27715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:14.791385 27715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:14.796043 27715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:14.796365 27715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:14.797521 27715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:14.798971 27715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:14.799139 27715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:14.799278 27715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:14.799381 27715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:14.835933 27715 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:46699
I20260806 09:15:14.836026 27901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:46699 every 8 connection(s)
I20260806 09:15:14.837419 27715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:15:14.839030 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:14.838007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:56401)
0806 09:15:14.838574 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:14.838723 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":286,"threads_started":1}
I20260806 09:15:14.842572 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:14.840731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:60687)
0806 09:15:14.841482 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:14.841703 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":299,"threads_started":1}
W20260806 09:15:14.843305 27902 heartbeater.cc:644] Failed to heartbeat to 127.3.69.62:45345 (0 consecutive failures): Network error: Failed to ping master at 127.3.69.62:45345: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
I20260806 09:15:14.844173 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:14.843808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:35055)
0806 09:15:14.843987 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:14.844065 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 09:15:14.845045 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:14.844728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:47885)
0806 09:15:14.844859 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:14.844935 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 09:15:14.845299 27902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:15:14.846799 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 27715
I20260806 09:15:15.332587 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:15.331714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:55139)
0806 09:15:15.332240 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:15.332354 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":265,"threads_started":1}
I20260806 09:15:15.840914 27910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:15.840188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:60449)
0806 09:15:15.840665 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:15.840754 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":237,"threads_started":1}
I20260806 09:15:15.846612 27911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:15.845855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:50589)
0806 09:15:15.846302 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:15.846388 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":217,"threads_started":1}
I20260806 09:15:16.334213 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:16.333458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:51453)
0806 09:15:16.333955 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:16.334037 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":261,"threads_started":1}
I20260806 09:15:16.842527 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:16.841794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:60287)
0806 09:15:16.842278 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:16.842357 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":260,"threads_started":1}
I20260806 09:15:16.848322 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:16.847616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:43671)
0806 09:15:16.848052 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:16.848139 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260806 09:15:17.335932 27915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:17.335148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:56335)
0806 09:15:17.335677 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:17.335753 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":294,"threads_started":1}
I20260806 09:15:17.844237 27916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:17.843531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:42135)
0806 09:15:17.843957 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:17.844039 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":206,"threads_started":1}
I20260806 09:15:17.849956 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:17.849157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:41199)
0806 09:15:17.849706 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:17.849785 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":323,"threads_started":1}
I20260806 09:15:18.337670 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:18.336909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:41191)
0806 09:15:18.337374 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:18.337460 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":243,"threads_started":1}
I20260806 09:15:18.845877 27921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:18.845152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:34029)
0806 09:15:18.845605 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:18.845703 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":233,"threads_started":1}
I20260806 09:15:18.851606 27923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:18.850820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:36739)
0806 09:15:18.851345 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:18.851425 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":301,"threads_started":1}
I20260806 09:15:19.339417 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:19.338597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:42165)
0806 09:15:19.339151 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:19.339236 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":255,"threads_started":1}
I20260806 09:15:19.847954 27926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:19.846837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:58809)
0806 09:15:19.847615 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:19.847714 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":506,"threads_started":1}
I20260806 09:15:19.853242 27927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:19.852519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:55233)
0806 09:15:19.852970 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:19.853070 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260806 09:15:20.341056 27928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:20.340314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:52595)
0806 09:15:20.340798 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:20.340885 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
I20260806 09:15:20.849810 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:20.848853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:42645)
0806 09:15:20.849412 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:20.849528 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":263,"threads_started":1}
I20260806 09:15:20.854772 27931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:20.854096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:42469)
0806 09:15:20.854534 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:20.854623 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260806 09:15:21.342988 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:21.341989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:45021)
0806 09:15:21.342662 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:21.342759 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":264,"threads_started":1}
I20260806 09:15:21.851531 27934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:21.850792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:57585)
0806 09:15:21.851286 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:21.851369 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":224,"threads_started":1}
I20260806 09:15:21.856513 27935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:21.855709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:40453)
0806 09:15:21.856201 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:21.856279 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":229,"threads_started":1}
I20260806 09:15:22.344693 27936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:22.343931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:40383)
0806 09:15:22.344405 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:22.344524 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":239,"threads_started":1}
I20260806 09:15:22.853325 27937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:22.852478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:46809)
0806 09:15:22.853063 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:22.853148 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":353,"threads_started":1}
I20260806 09:15:22.858131 27938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:22.857358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:59841)
0806 09:15:22.857859 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:22.857934 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":301,"threads_started":1}
I20260806 09:15:23.346238 27940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:23.345511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:42383)
0806 09:15:23.345973 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:23.346055 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":224,"threads_started":1}
I20260806 09:15:23.854993 27942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:23.854286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:36779)
0806 09:15:23.854732 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:23.854812 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":246,"threads_started":1}
I20260806 09:15:23.859870 27943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:23.858963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:58861)
0806 09:15:23.859573 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:23.859706 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":417,"threads_started":1}
I20260806 09:15:24.348297 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:24.347351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:38691)
0806 09:15:24.347929 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:24.348052 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":323,"threads_started":1}
I20260806 09:15:24.856698 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:24.855916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:35263)
0806 09:15:24.856410 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:24.856493 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
I20260806 09:15:24.861469 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:24.860796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:53055)
0806 09:15:24.861237 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:24.861314 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260806 09:15:25.350332 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:25.349453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:34235)
0806 09:15:25.349987 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:25.350134 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":323,"threads_started":1}
I20260806 09:15:25.858306 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:25.857582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:41379)
0806 09:15:25.858028 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:25.858109 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260806 09:15:25.863163 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:25.862339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:44949)
0806 09:15:25.862853 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:25.862939 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":299,"threads_started":1}
I20260806 09:15:26.352231 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:26.351367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:53415)
0806 09:15:26.351956 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:26.352035 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":348,"threads_started":1}
I20260806 09:15:26.860014 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:26.859224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:35633)
0806 09:15:26.859737 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:26.859824 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":224,"threads_started":1}
I20260806 09:15:26.864761 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:26.864050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:56461)
0806 09:15:26.864537 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:26.864614 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":271,"threads_started":1}
I20260806 09:15:27.353989 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:27.353214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:46329)
0806 09:15:27.353708 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:27.353795 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":230,"threads_started":1}
I20260806 09:15:27.861677 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:27.860918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:54393)
0806 09:15:27.861396 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:27.861487 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":240,"threads_started":1}
I20260806 09:15:27.866464 27968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:27.865622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:37471)
0806 09:15:27.866180 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:27.866270 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":301,"threads_started":1}
I20260806 09:15:28.355710 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:28.354875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:50839)
0806 09:15:28.355405 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:28.355495 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":300,"threads_started":1}
I20260806 09:15:28.863593 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:28.862846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:57021)
0806 09:15:28.863327 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:28.863415 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":277,"threads_started":1}
I20260806 09:15:28.868072 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:28.867321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:52857)
0806 09:15:28.867827 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:28.867902 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":283,"threads_started":1}
I20260806 09:15:29.357659 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:29.356827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.2:40839)
0806 09:15:29.357286 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:29.357369 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":224,"threads_started":1}
I20260806 09:15:29.853811 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 27261
I20260806 09:15:29.866050 27975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:29.864535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.1:52665)
0806 09:15:29.865465 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:29.865704 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":657,"threads_started":1}
I20260806 09:15:29.869904 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:29.869127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45345 (local address 127.3.69.3:40679)
0806 09:15:29.869641 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:29.869722 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.62:45345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":286,"threads_started":1}
I20260806 09:15:29.873883 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:29.855342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.3.69.1:46843)
0806 09:15:29.855985 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:29.856000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:29.856043 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:15:29.857716 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:29.857726 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:29.858198 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:29.858593 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:29.860326 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:29.861695 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:29.863137 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:29.863887 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:29.864118 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:29.867665 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:29.867673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:29.867688 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:29.867729 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:29.870901 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:29.871609 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:29.871616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:29.871622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:29.871786 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:29.872124 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:29.872128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:29.872130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:29.872653 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:29.872885 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:29.873252 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 09:15:29.873577 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":271,"threads_started":1}
I20260806 09:15:29.875289 27513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:29.960273 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 27261
I20260806 09:15:29.991958 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 27522
I20260806 09:15:30.010135 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:29.992983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.3.69.2:41509)
0806 09:15:29.993559 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:29.993573 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:29.993613 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:15:29.995201 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:29.995213 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:29.995632 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:29.995951 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:29.997236 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:29.998472 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:29.999835 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:30.000546 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:30.000701 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:30.003515 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:30.003525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:30.003541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:30.003588 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:30.006773 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:30.007569 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:30.007576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:30.007584 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:30.007776 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:30.008157 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:30.008161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:30.008163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:30.008718 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:30.009010 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:30.009442 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 09:15:30.009816 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260806 09:15:30.011665 27705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:30.097287 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 27522
I20260806 09:15:30.124832 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 27715
I20260806 09:15:30.143572 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:30.125894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.3.69.3:46699)
0806 09:15:30.126529 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:30.126543 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:30.126624 (+ 81us) server_negotiation.cc:407] Connection header received
0806 09:15:30.128433 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:30.128443 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:30.128901 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:30.129271 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:30.130829 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:30.132093 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:30.133506 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:30.134187 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:30.134409 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:30.137167 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:30.137175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:30.137189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:30.137238 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:30.140202 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:30.140900 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:30.140908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:30.140915 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:30.141095 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:30.141499 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:30.141504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:30.141507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:30.142028 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:30.142360 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:30.142811 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 09:15:30.143266 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":326,"threads_started":1}
I20260806 09:15:30.144961 27895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:30.231189 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 27715
2026-08-06T09:15:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65095 ms)
[----------] 30 tests from SecurityITest (311497 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:15:30 dist-test-slave-w4v5 krb5kdc[27988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:15:30 dist-test-slave-w4v5 krb5kdc[27988](info): set up 2 sockets
Aug 06 09:15:30 dist-test-slave-w4v5 krb5kdc[27988](info): commencing operation
krb5kdc: starting...
W20260806 09:15:31.567623 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:15:31 dist-test-slave-w4v5 krb5kdc[27988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:15:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:15:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:15:31.885042 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:33407
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:33407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:32.232864 28011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:32.233253 28011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:32.233439 28011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:32.240821 28011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:32.240988 28011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:32.241104 28011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:32.241212 28011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:32.241302 28011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:32.251183 28011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:33407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:32.252592 28011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:32.253955 28011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:32.266279 28018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:32.266227 28023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:32.266333 28017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:32.272461 28011 server_base.cc:1034] running on GCE node
I20260806 09:15:32.273370 28011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:32.274856 28011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:32.276089 28011 hybrid_clock.cc:648] HybridClock initialized: now 1786007732276023 us; error 94 us; skew 500 ppm
I20260806 09:15:32.276461 28011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:32 dist-test-slave-w4v5 krb5kdc[27988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007732, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:32.284848 28011 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:15:32.286571 28011 webserver.cc:460] Webserver started at http://127.3.69.62:44747/ using document root <none> and password file <none>
I20260806 09:15:32.287415 28011 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:32.287570 28011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:32.287889 28011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:32.290974 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf"
format_stamp: "Formatted at 2026-08-06 09:15:32 on dist-test-slave-w4v5"
server_key: "acdd70c2c63d97c6a32cef5c19d2d39f"
server_key_iv: "1a0250d36d992ca9ee9566b7ff7ada4f"
server_key_version: "encryptionkey@0"
I20260806 09:15:32.291813 28011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf"
format_stamp: "Formatted at 2026-08-06 09:15:32 on dist-test-slave-w4v5"
server_key: "acdd70c2c63d97c6a32cef5c19d2d39f"
server_key_iv: "1a0250d36d992ca9ee9566b7ff7ada4f"
server_key_version: "encryptionkey@0"
I20260806 09:15:32.297067 28011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:15:32.300570 28035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:32.301986 28011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 09:15:32.302217 28011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf"
format_stamp: "Formatted at 2026-08-06 09:15:32 on dist-test-slave-w4v5"
server_key: "acdd70c2c63d97c6a32cef5c19d2d39f"
server_key_iv: "1a0250d36d992ca9ee9566b7ff7ada4f"
server_key_version: "encryptionkey@0"
I20260806 09:15:32.302443 28011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:32.323282 28011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:32.327831 28011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:32.328135 28011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:32.347796 28011 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:33407
I20260806 09:15:32.347877 28110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:33407 every 8 connection(s)
I20260806 09:15:32.349222 28011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:32.353053 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:32.356549 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 28011
I20260806 09:15:32.356794 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:15:32.357638 3348 external_mini_cluster.cc:1428] Setting key 86f75ae8ec17bdec8906c57633f8f9b5
I20260806 09:15:32.366726 28111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf: Bootstrap starting.
I20260806 09:15:32.370579 28111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:32.371948 28111 log.cc:824] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:15:32 dist-test-slave-w4v5 krb5kdc[27988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:32.375226 28111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf: No bootstrap required, opened a new log
I20260806 09:15:32.383383 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:32.359913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.3.69.62:33407)
0806 09:15:32.360784 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:32.360795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:32.360836 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:15:32.362498 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:32.362522 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:32.362887 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:32.363300 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:32.365083 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:32.366375 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:32.368320 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:32.368994 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:32.369133 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:32.374339 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:32.374350 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:32.374371 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:32.374414 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:32.378620 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:32.379375 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:32.379383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:32.379391 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:32.379603 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:32.380119 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:32.380124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:32.380128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:32.380842 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:32.381163 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:32.381540 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 09:15:32.381990 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":434,"threads_started":1}
I20260806 09:15:32.385661 28111 raft_consensus.cc:348] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33407 } }
I20260806 09:15:32.386030 28111 raft_consensus.cc:374] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:32.386170 28111 raft_consensus.cc:729] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c966f4bc1a7842bb9b9b19bb3dd10fdf, State: Initialized, Role: FOLLOWER
I20260806 09:15:32.386714 28111 consensus_queue.cc:260] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33407 } }
I20260806 09:15:32.386956 28111 raft_consensus.cc:388] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:32.387130 28111 raft_consensus.cc:482] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:32.387328 28111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:32.389601 28111 raft_consensus.cc:504] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33407 } }
I20260806 09:15:32.390125 28111 leader_election.cc:302] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c966f4bc1a7842bb9b9b19bb3dd10fdf; no voters:
I20260806 09:15:32.390733 28111 leader_election.cc:288] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:15:32.390898 28117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:32.391718 28117 raft_consensus.cc:686] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [term 1 LEADER]: Becoming Leader. State: Replica: c966f4bc1a7842bb9b9b19bb3dd10fdf, State: Running, Role: LEADER
I20260806 09:15:32.392306 28117 consensus_queue.cc:237] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33407 } }
I20260806 09:15:32.392936 28111 sys_catalog.cc:564] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:32.395535 28120 sys_catalog.cc:455] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33407 } } }
I20260806 09:15:32.395994 28120 sys_catalog.cc:458] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:32.396627 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c966f4bc1a7842bb9b9b19bb3dd10fdf. Latest consensus state: current_term: 1 leader_uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c966f4bc1a7842bb9b9b19bb3dd10fdf" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33407 } } }
I20260806 09:15:32.396952 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:32.397559 28129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:32.403340 28129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:32.412846 28129 catalog_manager.cc:1347] Generated new cluster ID: 0643798b538a4782bf1c138430192ca8
I20260806 09:15:32.412976 28129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:32.442216 28129 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:15:32.443717 28129 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:32.450733 28129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf: Generated new TSK 0
I20260806 09:15:32.451606 28129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:15:32.539232 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:33407
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:32.919493 28149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:32.919792 28149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:32.919898 28149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:32.927176 28149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:32.927280 28149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:32.927393 28149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:15:32.936947 28149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:32.938187 28149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:32.939442 28149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:32.952448 28162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:32.952503 28158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:32.952466 28160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:32.953635 28149 server_base.cc:1034] running on GCE node
I20260806 09:15:32.954310 28149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:32.955257 28149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:32.956485 28149 hybrid_clock.cc:648] HybridClock initialized: now 1786007732956436 us; error 57 us; skew 500 ppm
I20260806 09:15:32.956804 28149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:32 dist-test-slave-w4v5 krb5kdc[27988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007732, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:32.960062 28149 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:15:32.961395 28149 webserver.cc:460] Webserver started at http://127.3.69.1:38725/ using document root <none> and password file <none>
I20260806 09:15:32.962015 28149 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:32.962116 28149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:32.962368 28149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:32.965193 28149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "83b2f1fe8fec413eada9ec14fe0a74d3"
format_stamp: "Formatted at 2026-08-06 09:15:32 on dist-test-slave-w4v5"
server_key: "7b57f63d1ee0e0b1b26436cdec424159"
server_key_iv: "3877f3c8d1c6cb5f58f3b583e151a20e"
server_key_version: "encryptionkey@0"
I20260806 09:15:32.965922 28149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "83b2f1fe8fec413eada9ec14fe0a74d3"
format_stamp: "Formatted at 2026-08-06 09:15:32 on dist-test-slave-w4v5"
server_key: "7b57f63d1ee0e0b1b26436cdec424159"
server_key_iv: "3877f3c8d1c6cb5f58f3b583e151a20e"
server_key_version: "encryptionkey@0"
I20260806 09:15:32.970851 28149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:15:32.974169 28174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:32.975500 28149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:15:32.975627 28149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "83b2f1fe8fec413eada9ec14fe0a74d3"
format_stamp: "Formatted at 2026-08-06 09:15:32 on dist-test-slave-w4v5"
server_key: "7b57f63d1ee0e0b1b26436cdec424159"
server_key_iv: "3877f3c8d1c6cb5f58f3b583e151a20e"
server_key_version: "encryptionkey@0"
I20260806 09:15:32.975811 28149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:32.993872 28149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:32.998483 28149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:32.998813 28149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:32.999914 28149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:33.001314 28149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:33.001420 28149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:33.001577 28149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:33.001660 28149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:33.037021 28149 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:38175
I20260806 09:15:33.037112 28346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:38175 every 8 connection(s)
I20260806 09:15:33.038355 28149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:15:33.047494 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 28149
I20260806 09:15:33.047748 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:15:33.048641 3348 external_mini_cluster.cc:1428] Setting key 517ddc1734caca9b984e1ce7c6686b73
Aug 06 09:15:33 dist-test-slave-w4v5 krb5kdc[27988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007732, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:33.060727 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:33.041043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:48899 (local address 127.3.69.62:33407)
0806 09:15:33.041482 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:33.041486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:33.042455 (+ 969us) server_negotiation.cc:407] Connection header received
0806 09:15:33.044391 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:33.044396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:33.044467 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:33.044568 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:33.045947 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:33.046661 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:33.049026 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:33.049420 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:33.049571 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:33.053725 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:33.053734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:33.053741 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:33.053783 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:33.057374 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:33.058087 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:33.058094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:33.058100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:33.058265 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:33.058709 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:33.058717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:33.058723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:33.059124 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:33.059276 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:33.059975 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 09:15:33.060236 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260806 09:15:33.061290 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:33.041606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:33407 (local address 127.3.69.1:48899)
0806 09:15:33.042298 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:33.042335 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:33.044140 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:33.044766 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:33.044773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:33.045275 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:33.045801 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:33.045817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:33.046803 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:33.046807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:33.048882 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:33.048899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:33.049648 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:33.049653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:33.049938 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:33.050835 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:33.050877 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:33.053493 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:33.057520 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:33.057527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:33.057538 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:33.057951 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:33.058375 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:33.058378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:33.058380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:33.058555 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:33.059248 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:33.059254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:33.059544 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 09:15:33.059798 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 09:15:33.060160 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":271,"threads_started":1}
I20260806 09:15:33.062929 28347 heartbeater.cc:347] Connected to a master server at 127.3.69.62:33407
I20260806 09:15:33.063252 28347 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:33.063980 28347 heartbeater.cc:511] Master 127.3.69.62:33407 requested a full tablet report, sending...
I20260806 09:15:33.066169 28058 ts_manager.cc:194] Registered new tserver with Master: 83b2f1fe8fec413eada9ec14fe0a74d3 (127.3.69.1:38175)
I20260806 09:15:33.067730 28058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:48899
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:15:33.118628 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:33407
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:33.484144 28358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:33.484457 28358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:33.484620 28358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:33.492408 28358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:33.492558 28358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:33.492722 28358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:15:33.502871 28358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:33.504194 28358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:33.505499 28358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:33.518744 28368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:33.518699 28370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:33.518754 28372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:33.519764 28358 server_base.cc:1034] running on GCE node
I20260806 09:15:33.520385 28358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:33.521528 28358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:33.522796 28358 hybrid_clock.cc:648] HybridClock initialized: now 1786007733522724 us; error 43 us; skew 500 ppm
I20260806 09:15:33.523264 28358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:33 dist-test-slave-w4v5 krb5kdc[27988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007733, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:33.526819 28358 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:15:33.528416 28358 webserver.cc:460] Webserver started at http://127.3.69.2:34903/ using document root <none> and password file <none>
I20260806 09:15:33.529086 28358 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:33.529191 28358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:33.529455 28358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:33.532493 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "5e54f9bfc5b04fab9376b33ff2420702"
format_stamp: "Formatted at 2026-08-06 09:15:33 on dist-test-slave-w4v5"
server_key: "1990c5b5ec78576b7fa164eec8c117b4"
server_key_iv: "acc1686014c31f530627fc7c4080e312"
server_key_version: "encryptionkey@0"
I20260806 09:15:33.533283 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "5e54f9bfc5b04fab9376b33ff2420702"
format_stamp: "Formatted at 2026-08-06 09:15:33 on dist-test-slave-w4v5"
server_key: "1990c5b5ec78576b7fa164eec8c117b4"
server_key_iv: "acc1686014c31f530627fc7c4080e312"
server_key_version: "encryptionkey@0"
I20260806 09:15:33.538450 28358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 09:15:33.541978 28385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:33.543354 28358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 09:15:33.543536 28358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "5e54f9bfc5b04fab9376b33ff2420702"
format_stamp: "Formatted at 2026-08-06 09:15:33 on dist-test-slave-w4v5"
server_key: "1990c5b5ec78576b7fa164eec8c117b4"
server_key_iv: "acc1686014c31f530627fc7c4080e312"
server_key_version: "encryptionkey@0"
I20260806 09:15:33.543743 28358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:33.576167 28358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:33.580336 28358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:33.580639 28358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:33.581678 28358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:33.583158 28358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:33.583231 28358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:33.583389 28358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:33.583438 28358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:33.618216 28358 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:35863
I20260806 09:15:33.618294 28551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:35863 every 8 connection(s)
I20260806 09:15:33.619616 28358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:15:33.620602 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 28358
I20260806 09:15:33.620806 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:15:33.621412 3348 external_mini_cluster.cc:1428] Setting key 33baef9fc6527d41558b4ec4e2eb3d9e
Aug 06 09:15:33 dist-test-slave-w4v5 krb5kdc[27988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007733, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:33.643951 28554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:33.622564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:55531 (local address 127.3.69.62:33407)
0806 09:15:33.623177 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:33.623183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:33.624269 (+ 1086us) server_negotiation.cc:407] Connection header received
0806 09:15:33.627260 (+ 2991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:33.627265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:33.627333 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:33.627421 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:33.629543 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:33.630296 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:33.632830 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:33.633294 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:33.633392 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:33.637598 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:33.637607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:33.637614 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:33.637661 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:33.640591 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:33.641530 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:33.641534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:33.641536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:33.641663 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:33.642164 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:33.642167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:33.642169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:33.642424 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:33.642539 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:33.643487 (+ 948us) server_negotiation.cc:299] Negotiation successful
0806 09:15:33.643656 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":292,"threads_started":1}
I20260806 09:15:33.645577 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:33.623268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:33407 (local address 127.3.69.2:55531)
0806 09:15:33.624077 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:33.624137 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:33.626930 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:33.627703 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:33.627717 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:33.628519 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:33.629355 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:33.629383 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:33.630460 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:33.630468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:33.632653 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:33.632671 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:33.633538 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:33.633547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:33.633895 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:33.634634 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:33.634675 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:33.637369 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:33.640737 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:33.640748 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:33.640767 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:33.641404 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:33.641790 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:33.641797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:33.641803 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:33.642048 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:33.642565 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:33.642576 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:33.643045 (+ 469us) client_negotiation.cc:769] Sending connection context
0806 09:15:33.643388 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 09:15:33.643974 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":343,"threads_started":1}
I20260806 09:15:33.647068 28552 heartbeater.cc:347] Connected to a master server at 127.3.69.62:33407
I20260806 09:15:33.647469 28552 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:33.648283 28552 heartbeater.cc:511] Master 127.3.69.62:33407 requested a full tablet report, sending...
I20260806 09:15:33.649765 28058 ts_manager.cc:194] Registered new tserver with Master: 5e54f9bfc5b04fab9376b33ff2420702 (127.3.69.2:35863)
I20260806 09:15:33.650614 28058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:55531
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:15:33.696326 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:33407
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:34.044431 28565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:34.044744 28565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:34.044919 28565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:34.052354 28565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:34.052459 28565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:34.052573 28565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:15:34.062690 28565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:34.064095 28565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:34.065377 28565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 09:15:34.070524 28347 heartbeater.cc:503] Master 127.3.69.62:33407 was elected leader, sending a full tablet report...
W20260806 09:15:34.078778 28578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:34.078903 28576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:34.079288 28573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:34.079908 28565 server_base.cc:1034] running on GCE node
I20260806 09:15:34.080559 28565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:34.081447 28565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:34.082679 28565 hybrid_clock.cc:648] HybridClock initialized: now 1786007734082647 us; error 47 us; skew 500 ppm
I20260806 09:15:34.083004 28565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:34 dist-test-slave-w4v5 krb5kdc[27988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007734, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:34.086079 28565 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:15:34.087569 28565 webserver.cc:460] Webserver started at http://127.3.69.3:42387/ using document root <none> and password file <none>
I20260806 09:15:34.088234 28565 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:34.088375 28565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:34.088713 28565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:34.091609 28565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "bae7f61054404aeba319efc7b34f6011"
format_stamp: "Formatted at 2026-08-06 09:15:34 on dist-test-slave-w4v5"
server_key: "d59d675ce8b605c4cff5a1276c3d8c92"
server_key_iv: "82a96750121b89bc743274c4f0e22604"
server_key_version: "encryptionkey@0"
I20260806 09:15:34.092404 28565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "bae7f61054404aeba319efc7b34f6011"
format_stamp: "Formatted at 2026-08-06 09:15:34 on dist-test-slave-w4v5"
server_key: "d59d675ce8b605c4cff5a1276c3d8c92"
server_key_iv: "82a96750121b89bc743274c4f0e22604"
server_key_version: "encryptionkey@0"
I20260806 09:15:34.097669 28565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 09:15:34.101199 28587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:34.102434 28565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:15:34.102607 28565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "bae7f61054404aeba319efc7b34f6011"
format_stamp: "Formatted at 2026-08-06 09:15:34 on dist-test-slave-w4v5"
server_key: "d59d675ce8b605c4cff5a1276c3d8c92"
server_key_iv: "82a96750121b89bc743274c4f0e22604"
server_key_version: "encryptionkey@0"
I20260806 09:15:34.102823 28565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:34.117766 28565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:34.121635 28565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:34.121925 28565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:34.122854 28565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:34.124186 28565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:34.124292 28565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:34.124434 28565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:34.124521 28565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:34.158818 28565 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:40009
I20260806 09:15:34.158854 28751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:40009 every 8 connection(s)
I20260806 09:15:34.160166 28565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:15:34.167836 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 28565
I20260806 09:15:34.168021 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:15:34.168675 3348 external_mini_cluster.cc:1428] Setting key ffb74d76c29c2feee5df8b0d4617a6b8
Aug 06 09:15:34 dist-test-slave-w4v5 krb5kdc[27988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007734, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:34.181260 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.162785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:56941 (local address 127.3.69.62:33407)
0806 09:15:34.163323 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.163328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.164266 (+ 938us) server_negotiation.cc:407] Connection header received
0806 09:15:34.166537 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.166543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.166615 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.166739 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:34.168401 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.169567 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.171793 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.172303 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.172404 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.175692 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:34.175697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:34.175700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:34.175733 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:34.178410 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.179287 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.179291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.179293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.179413 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.179788 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.179791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.179793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.180040 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:34.180214 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.180824 (+ 610us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.180982 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":277,"threads_started":1}
I20260806 09:15:34.182260 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.163388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:33407 (local address 127.3.69.3:56941)
0806 09:15:34.164089 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:34.164131 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:34.166221 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:34.166941 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:34.166949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:34.167538 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:34.168221 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.168242 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.169752 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.169758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.171653 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.171669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.172500 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.172506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.172705 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.173321 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:34.173342 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:34.175502 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:34.178607 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:34.178613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:34.178624 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:34.179160 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:34.179508 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:34.179511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:34.179513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:34.179685 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:34.180209 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:34.180215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:34.180499 (+ 284us) client_negotiation.cc:769] Sending connection context
0806 09:15:34.180717 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 09:15:34.181106 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":291,"threads_started":1}
I20260806 09:15:34.183825 28752 heartbeater.cc:347] Connected to a master server at 127.3.69.62:33407
I20260806 09:15:34.184165 28752 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:34.184890 28752 heartbeater.cc:511] Master 127.3.69.62:33407 requested a full tablet report, sending...
I20260806 09:15:34.186328 28058 ts_manager.cc:194] Registered new tserver with Master: bae7f61054404aeba319efc7b34f6011 (127.3.69.3:40009)
I20260806 09:15:34.187239 28058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:56941
I20260806 09:15:34.196584 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:15:34.214733 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.201564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.3.69.62:33407)
0806 09:15:34.201768 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.201772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.202241 (+ 469us) server_negotiation.cc:407] Connection header received
0806 09:15:34.202423 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.202427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.202491 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.202619 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:34.204190 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.204950 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.206536 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.207076 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.207226 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.209700 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:34.209705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:34.209708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:34.209741 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:34.212301 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.212918 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.212922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.212924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.213042 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.213365 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.213368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.213370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.213685 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:34.213876 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.214287 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.214466 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 09:15:34.218828 28058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:15:34.225917 28058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:15:34.256915 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.237842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38175 (local address 127.0.0.1:40912)
0806 09:15:34.244939 (+ 7097us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:34.244991 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:34.245217 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:34.249376 (+ 4159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:34.249384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:34.249413 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:34.249985 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.250004 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.252098 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.252104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.254364 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.254377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.255834 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.255839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.256040 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.256108 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 09:15:34.256222 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 09:15:34.256354 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6752,"mutex_wait_us":3,"thread_start_us":10892,"threads_started":1}
I20260806 09:15:34.257591 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.238035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:35863 (local address 127.0.0.1:58600)
0806 09:15:34.247754 (+ 9719us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:34.247775 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:34.247968 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:34.249713 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:34.249718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:34.249740 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:34.250330 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.250347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.252492 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.252497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.254676 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.254692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.256726 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.256732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.256937 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.256975 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:15:34.257040 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 09:15:34.257190 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9593,"mutex_wait_us":30,"thread_start_us":6359,"threads_started":1}
I20260806 09:15:34.257738 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.238018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.3.69.1:38175)
0806 09:15:34.247801 (+ 9783us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.247813 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.247845 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:15:34.248583 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.248593 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.248880 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.249085 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:34.250245 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.251903 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.254541 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.255589 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.256872 (+ 1283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.257146 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.257225 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.257400 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9604,"thread_start_us":9071,"threads_started":1}
I20260806 09:15:34.258548 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.238693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.3.69.2:35863)
0806 09:15:34.249112 (+ 10419us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.249121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.249149 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:15:34.249255 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.249262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.249478 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.250592 (+ 1114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:34.250836 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.252337 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.254914 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.256471 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.257722 (+ 1251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.257932 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.258014 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.258202 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10257,"thread_start_us":9625,"threads_started":1}
I20260806 09:15:34.259270 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.245628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:40009 (local address 127.0.0.1:54624)
0806 09:15:34.250634 (+ 5006us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:34.250653 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:34.250779 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:34.251364 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:34.251369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:34.251395 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:34.251743 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.251756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.253746 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.253751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.256542 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.256559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.258402 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.258409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.258598 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.258631 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:15:34.258733 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 09:15:34.258867 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4881,"mutex_wait_us":26,"thread_start_us":4713,"threads_started":1}
I20260806 09:15:34.259589 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.245641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.3.69.3:40009)
0806 09:15:34.248811 (+ 3170us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.248819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.250772 (+ 1953us) server_negotiation.cc:407] Connection header received
0806 09:15:34.250915 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.250922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.251157 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.251350 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:34.251937 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.253593 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.256744 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.258198 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.258331 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.258537 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.259085 (+ 548us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.259282 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3013,"thread_start_us":2845,"threads_started":1}
I20260806 09:15:34.261952 28250 tablet_service.cc:1467] Processing CreateTablet for tablet ecc83749954e4cc6aae694ebbd297a27 (DEFAULT_TABLE table=test-table [id=e7257aaceb9d4ca6b9997f33842e99ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:34.262217 28454 tablet_service.cc:1467] Processing CreateTablet for tablet ecc83749954e4cc6aae694ebbd297a27 (DEFAULT_TABLE table=test-table [id=e7257aaceb9d4ca6b9997f33842e99ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:34.263324 28250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc83749954e4cc6aae694ebbd297a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:34.263015 28652 tablet_service.cc:1467] Processing CreateTablet for tablet ecc83749954e4cc6aae694ebbd297a27 (DEFAULT_TABLE table=test-table [id=e7257aaceb9d4ca6b9997f33842e99ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:34.263633 28454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc83749954e4cc6aae694ebbd297a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:34.264423 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc83749954e4cc6aae694ebbd297a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:34.275512 28786 tablet_bootstrap.cc:492] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: Bootstrap starting.
I20260806 09:15:34.279531 28786 tablet_bootstrap.cc:654] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:34.279775 28787 tablet_bootstrap.cc:492] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Bootstrap starting.
I20260806 09:15:34.280365 28789 tablet_bootstrap.cc:492] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: Bootstrap starting.
I20260806 09:15:34.281374 28786 log.cc:824] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:34.283581 28787 tablet_bootstrap.cc:654] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:34.283833 28789 tablet_bootstrap.cc:654] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:34.284771 28786 tablet_bootstrap.cc:492] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: No bootstrap required, opened a new log
I20260806 09:15:34.284871 28789 log.cc:824] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:34.284904 28787 log.cc:824] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:34.285055 28786 ts_tablet_manager.cc:1397] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 09:15:34.288331 28787 tablet_bootstrap.cc:492] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: No bootstrap required, opened a new log
I20260806 09:15:34.288606 28787 ts_tablet_manager.cc:1397] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 09:15:34.289389 28789 tablet_bootstrap.cc:492] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: No bootstrap required, opened a new log
I20260806 09:15:34.289656 28789 ts_tablet_manager.cc:1397] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 09:15:34.295965 28786 raft_consensus.cc:348] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.296463 28786 raft_consensus.cc:374] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:34.296568 28786 raft_consensus.cc:729] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b2f1fe8fec413eada9ec14fe0a74d3, State: Initialized, Role: FOLLOWER
I20260806 09:15:34.297219 28789 raft_consensus.cc:348] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.297235 28786 consensus_queue.cc:260] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.297573 28789 raft_consensus.cc:374] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:34.297981 28789 raft_consensus.cc:729] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae7f61054404aeba319efc7b34f6011, State: Initialized, Role: FOLLOWER
I20260806 09:15:34.298898 28789 consensus_queue.cc:260] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.299790 28787 raft_consensus.cc:348] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.300204 28787 raft_consensus.cc:374] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:34.300313 28787 raft_consensus.cc:729] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e54f9bfc5b04fab9376b33ff2420702, State: Initialized, Role: FOLLOWER
I20260806 09:15:34.300547 28786 ts_tablet_manager.cc:1428] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: Time spent starting tablet: real 0.015s user 0.010s sys 0.007s
I20260806 09:15:34.300890 28787 consensus_queue.cc:260] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.301671 28752 heartbeater.cc:503] Master 127.3.69.62:33407 was elected leader, sending a full tablet report...
I20260806 09:15:34.302414 28789 ts_tablet_manager.cc:1428] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260806 09:15:34.303378 28552 heartbeater.cc:503] Master 127.3.69.62:33407 was elected leader, sending a full tablet report...
I20260806 09:15:34.303977 28787 ts_tablet_manager.cc:1428] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260806 09:15:34.307288 28796 raft_consensus.cc:482] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:15:34.307628 28796 raft_consensus.cc:504] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.309470 28796 leader_election.cc:288] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b2f1fe8fec413eada9ec14fe0a74d3 (127.3.69.1:38175), 5e54f9bfc5b04fab9376b33ff2420702 (127.3.69.2:35863)
I20260806 09:15:34.315939 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.309811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:38175 (local address 127.3.69.3:60595)
0806 09:15:34.310129 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:34.310145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:34.310296 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:34.310677 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:34.310684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:34.310719 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:34.310945 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.310960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.312259 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.312264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.313857 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.313869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.315385 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.315389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.315549 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.315574 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:15:34.315654 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 09:15:34.315742 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":35}
I20260806 09:15:34.316087 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.309707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:60595 (local address 127.3.69.1:38175)
0806 09:15:34.309915 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.309923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.310228 (+ 305us) server_negotiation.cc:407] Connection header received
0806 09:15:34.310431 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.310436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.310531 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.310641 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:34.311133 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.312129 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.314015 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.315240 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.315376 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.315477 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.315708 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.315838 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 09:15:34.316926 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.309899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:35183 (local address 127.3.69.2:35863)
0806 09:15:34.310587 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.310595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.310767 (+ 172us) server_negotiation.cc:407] Connection header received
0806 09:15:34.311102 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.311110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.311186 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.311294 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:34.311853 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.312802 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.314813 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.316055 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.316209 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.316308 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.316667 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.316761 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526}
I20260806 09:15:34.317157 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.310035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:35863 (local address 127.3.69.3:35183)
0806 09:15:34.310652 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:34.310668 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:34.310858 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:34.311308 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:34.311312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:34.311337 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:34.311687 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.311700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.312961 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.312965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.314658 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:34.314670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.316335 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.316341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:34.316464 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.316527 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 09:15:34.316636 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 09:15:34.316772 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":46,"thread_start_us":331,"threads_started":1}
I20260806 09:15:34.317013 28283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc83749954e4cc6aae694ebbd297a27" candidate_uuid: "bae7f61054404aeba319efc7b34f6011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" is_pre_election: true
I20260806 09:15:34.317559 28283 raft_consensus.cc:2454] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae7f61054404aeba319efc7b34f6011 in term 0.
I20260806 09:15:34.318097 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc83749954e4cc6aae694ebbd297a27" candidate_uuid: "bae7f61054404aeba319efc7b34f6011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e54f9bfc5b04fab9376b33ff2420702" is_pre_election: true
I20260806 09:15:34.318315 28588 leader_election.cc:302] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b2f1fe8fec413eada9ec14fe0a74d3, bae7f61054404aeba319efc7b34f6011; no voters:
I20260806 09:15:34.318478 28483 raft_consensus.cc:2454] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae7f61054404aeba319efc7b34f6011 in term 0.
I20260806 09:15:34.318814 28796 raft_consensus.cc:2781] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:15:34.318991 28796 raft_consensus.cc:482] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:15:34.319136 28796 raft_consensus.cc:3037] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:34.321385 28796 raft_consensus.cc:504] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.321980 28796 leader_election.cc:288] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [CANDIDATE]: Term 1 election: Requested vote from peers 83b2f1fe8fec413eada9ec14fe0a74d3 (127.3.69.1:38175), 5e54f9bfc5b04fab9376b33ff2420702 (127.3.69.2:35863)
I20260806 09:15:34.322505 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc83749954e4cc6aae694ebbd297a27" candidate_uuid: "bae7f61054404aeba319efc7b34f6011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e54f9bfc5b04fab9376b33ff2420702"
I20260806 09:15:34.322497 28283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc83749954e4cc6aae694ebbd297a27" candidate_uuid: "bae7f61054404aeba319efc7b34f6011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3"
I20260806 09:15:34.322714 28283 raft_consensus.cc:3037] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:34.322714 28483 raft_consensus.cc:3037] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:34.326076 28283 raft_consensus.cc:2454] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae7f61054404aeba319efc7b34f6011 in term 1.
I20260806 09:15:34.326085 28483 raft_consensus.cc:2454] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae7f61054404aeba319efc7b34f6011 in term 1.
I20260806 09:15:34.326507 28591 leader_election.cc:302] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e54f9bfc5b04fab9376b33ff2420702, bae7f61054404aeba319efc7b34f6011; no voters:
I20260806 09:15:34.326870 28796 raft_consensus.cc:2781] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:34.327067 28796 raft_consensus.cc:686] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 1 LEADER]: Becoming Leader. State: Replica: bae7f61054404aeba319efc7b34f6011, State: Running, Role: LEADER
I20260806 09:15:34.327666 28796 consensus_queue.cc:237] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } }
I20260806 09:15:34.332101 28056 catalog_manager.cc:5495] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 reported cstate change: term changed from 0 to 1, leader changed from <none> to bae7f61054404aeba319efc7b34f6011 (127.3.69.3). New cstate: current_term: 1 leader_uuid: "bae7f61054404aeba319efc7b34f6011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae7f61054404aeba319efc7b34f6011" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 40009 } health_report { overall_health: HEALTHY } } }
I20260806 09:15:34.360257 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.354618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.3.69.3:40009)
0806 09:15:34.354826 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.354834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.354993 (+ 159us) server_negotiation.cc:407] Connection header received
0806 09:15:34.355251 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.355256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.355341 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.355496 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:15:34.355907 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.356613 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.358055 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.358461 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.358605 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.359171 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:15:34.359652 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:15:34.359825 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.359909 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.359988 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 09:15:34.368242 28483 raft_consensus.cc:1261] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 1 FOLLOWER]: Refusing update from remote peer bae7f61054404aeba319efc7b34f6011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:34.368240 28283 raft_consensus.cc:1261] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 1 FOLLOWER]: Refusing update from remote peer bae7f61054404aeba319efc7b34f6011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:34.369331 28796 consensus_queue.cc:1035] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b2f1fe8fec413eada9ec14fe0a74d3" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 38175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:34.369648 28798 consensus_queue.cc:1035] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e54f9bfc5b04fab9376b33ff2420702" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 35863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:15:34.371659 28553 tablet.cc:2369] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:34.381156 28802 mvcc.cc:205] Tried to move back new op lower bound from 7315487679961620480 to 7315487679810519040. Current Snapshot: MvccSnapshot[applied={T|T < 7315487679961620480}]
I20260806 09:15:34.381836 28804 mvcc.cc:205] Tried to move back new op lower bound from 7315487679961620480 to 7315487679810519040. Current Snapshot: MvccSnapshot[applied={T|T < 7315487679961620480}]
I20260806 09:15:34.396423 28058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:34.396721 28058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:34.400812 28058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c966f4bc1a7842bb9b9b19bb3dd10fdf: Sending DeleteTablet for 3 replicas of tablet ecc83749954e4cc6aae694ebbd297a27
I20260806 09:15:34.402210 28454 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc83749954e4cc6aae694ebbd297a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:34 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:58600
I20260806 09:15:34.402211 28250 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc83749954e4cc6aae694ebbd297a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:34 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:40912
I20260806 09:15:34.403155 28817 tablet_replica.cc:330] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: stopping tablet replica
I20260806 09:15:34.403301 28652 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc83749954e4cc6aae694ebbd297a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:34 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:54624
I20260806 09:15:34.403729 28817 raft_consensus.cc:2229] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:34.404309 28817 raft_consensus.cc:2258] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:34.405174 28813 tablet_replica.cc:330] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: stopping tablet replica
I20260806 09:15:34.405634 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 28149
I20260806 09:15:34.405671 28813 raft_consensus.cc:2229] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:34.406289 28813 raft_consensus.cc:2258] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:34.409160 28813 ts_tablet_manager.cc:1905] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:34.409953 28817 ts_tablet_manager.cc:1905] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:34.414608 28818 tablet_replica.cc:330] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: stopping tablet replica
I20260806 09:15:34.415107 28818 raft_consensus.cc:2229] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:15:34.415940 28818 raft_consensus.cc:2258] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:34.417300 28813 ts_tablet_manager.cc:1918] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:34.417470 28813 log.cc:1196] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/ecc83749954e4cc6aae694ebbd297a27
I20260806 09:15:34.418036 28813 ts_tablet_manager.cc:1939] T ecc83749954e4cc6aae694ebbd297a27 P 5e54f9bfc5b04fab9376b33ff2420702: Deleting consensus metadata
I20260806 09:15:34.418370 28818 ts_tablet_manager.cc:1905] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:34.419971 28041 catalog_manager.cc:4841] TS 5e54f9bfc5b04fab9376b33ff2420702 (127.3.69.2:35863): tablet ecc83749954e4cc6aae694ebbd297a27 (table test-table [id=e7257aaceb9d4ca6b9997f33842e99ec]) successfully deleted
I20260806 09:15:34.425276 28817 ts_tablet_manager.cc:1918] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:34.425462 28817 log.cc:1196] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/ecc83749954e4cc6aae694ebbd297a27
I20260806 09:15:34.426021 28817 ts_tablet_manager.cc:1939] T ecc83749954e4cc6aae694ebbd297a27 P 83b2f1fe8fec413eada9ec14fe0a74d3: Deleting consensus metadata
I20260806 09:15:34.427935 28038 catalog_manager.cc:4841] TS 83b2f1fe8fec413eada9ec14fe0a74d3 (127.3.69.1:38175): tablet ecc83749954e4cc6aae694ebbd297a27 (table test-table [id=e7257aaceb9d4ca6b9997f33842e99ec]) successfully deleted
I20260806 09:15:34.428344 28818 ts_tablet_manager.cc:1918] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:34.428519 28818 log.cc:1196] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/ecc83749954e4cc6aae694ebbd297a27
I20260806 09:15:34.429124 28818 ts_tablet_manager.cc:1939] T ecc83749954e4cc6aae694ebbd297a27 P bae7f61054404aeba319efc7b34f6011: Deleting consensus metadata
I20260806 09:15:34.430550 28041 catalog_manager.cc:4841] TS bae7f61054404aeba319efc7b34f6011 (127.3.69.3:40009): tablet ecc83749954e4cc6aae694ebbd297a27 (table test-table [id=e7257aaceb9d4ca6b9997f33842e99ec]) successfully deleted
Aug 06 09:15:34 dist-test-slave-w4v5 krb5kdc[27988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:15:34.444362 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.407325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.3.69.1:38175)
0806 09:15:34.407540 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.407549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.428261 (+ 20712us) server_negotiation.cc:407] Connection header received
0806 09:15:34.428602 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.428611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.428710 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.428833 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:34.430573 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.431433 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.433321 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.433723 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.433815 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.438596 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:34.438608 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:34.438624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:34.438665 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:34.441530 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.442199 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.442213 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.442216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.442347 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.442732 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.442738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.442741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.443166 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:34.443481 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.443778 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.443995 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 09:15:34.445107 28342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:34.537185 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 28149
I20260806 09:15:34.563995 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 28358
Aug 06 09:15:34 dist-test-slave-w4v5 krb5kdc[27988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:15:34.581873 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.565006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.3.69.2:35863)
0806 09:15:34.565237 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.565244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.565267 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:15:34.565337 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.565343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.565431 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.565554 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:34.566951 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.568182 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.569517 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.570154 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.570288 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.574868 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:34.574889 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:34.574911 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:34.574957 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:34.578884 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.579633 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.579652 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.579656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.579816 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.580171 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.580177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.580180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.580683 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:34.581013 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.581249 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.581481 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:15:34.582669 28544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:34.674180 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 28358
I20260806 09:15:34.706419 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 28565
Aug 06 09:15:34 dist-test-slave-w4v5 krb5kdc[27988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:15:34.725059 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:34.707515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.3.69.3:40009)
0806 09:15:34.707739 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:34.707747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:34.707778 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:15:34.707903 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:34.707911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:34.708028 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:34.708154 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:34.709490 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.710781 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.712170 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:34.712879 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:34.713004 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:34.717769 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:34.717786 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:34.717808 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:34.717855 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:34.721970 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.722687 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.722697 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.722700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.722920 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:34.723345 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:34.723352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:34.723361 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:34.723865 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:34.724212 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:34.724404 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 09:15:34.724642 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 09:15:34.725961 28745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:34.857462 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 28565
I20260806 09:15:34.886711 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 28011
I20260806 09:15:34.888129 28103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:35.001274 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 28011
2026-08-06T09:15:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4779 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T09:15:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:15:35Z Disabled control of system clock
I20260806 09:15:35.111629 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40309
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:40309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:15:35.463264 28832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:35.463554 28832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:35.463644 28832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:35.470861 28832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:35.470960 28832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:35.470999 28832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:35.471055 28832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:35.471161 28832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:35.480194 28832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:40309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:35.481509 28832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:35.483390 28832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:35.495481 28840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:35.495612 28845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:35.495682 28842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:35.496958 28832 server_base.cc:1034] running on GCE node
I20260806 09:15:35.497795 28832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:35.499183 28832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:35.500429 28832 hybrid_clock.cc:648] HybridClock initialized: now 1786007735500378 us; error 55 us; skew 500 ppm
I20260806 09:15:35.500794 28832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:35.503011 28832 webserver.cc:460] Webserver started at http://127.3.69.62:37805/ using document root <none> and password file <none>
I20260806 09:15:35.503726 28832 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:35.503860 28832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:35.504182 28832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:35.507066 28832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "669ea0ba673040508846c6bfa94092aa"
format_stamp: "Formatted at 2026-08-06 09:15:35 on dist-test-slave-w4v5"
server_key: "e3defe2a17465e515cba4c70df3be56c"
server_key_iv: "545bb3f9ce708e16ae38fd7add4b84f5"
server_key_version: "encryptionkey@0"
I20260806 09:15:35.507957 28832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "669ea0ba673040508846c6bfa94092aa"
format_stamp: "Formatted at 2026-08-06 09:15:35 on dist-test-slave-w4v5"
server_key: "e3defe2a17465e515cba4c70df3be56c"
server_key_iv: "545bb3f9ce708e16ae38fd7add4b84f5"
server_key_version: "encryptionkey@0"
I20260806 09:15:35.513219 28832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:15:35.516773 28852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:35.518229 28832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:15:35.518441 28832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "669ea0ba673040508846c6bfa94092aa"
format_stamp: "Formatted at 2026-08-06 09:15:35 on dist-test-slave-w4v5"
server_key: "e3defe2a17465e515cba4c70df3be56c"
server_key_iv: "545bb3f9ce708e16ae38fd7add4b84f5"
server_key_version: "encryptionkey@0"
I20260806 09:15:35.518683 28832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:35.544382 28832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:35.545298 28832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:35.545609 28832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:35.565532 28832 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:40309
I20260806 09:15:35.565554 28931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:40309 every 8 connection(s)
I20260806 09:15:35.566951 28832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:35.570933 28932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:35.572252 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 28832
I20260806 09:15:35.572445 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:15:35.573030 3348 external_mini_cluster.cc:1428] Setting key c9f4d4003d6c747b7690665af511cf46
I20260806 09:15:35.584317 28932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa: Bootstrap starting.
I20260806 09:15:35.586830 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:35.575121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.3.69.62:40309)
0806 09:15:35.576156 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:35.576175 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:35.576228 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:15:35.577910 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:35.577933 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:35.578422 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:35.578834 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:35.579454 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:35.580712 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:35.582262 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:35.582931 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:35.583165 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:35.584523 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:35.584535 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:35.584556 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:35.585102 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:35.585189 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:35.585210 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:35.585342 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:35.585569 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 09:15:35.585901 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":387,"threads_started":1}
I20260806 09:15:35.587777 28932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:35.588897 28932 log.cc:824] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:35.591759 28932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa: No bootstrap required, opened a new log
I20260806 09:15:35.598899 28932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669ea0ba673040508846c6bfa94092aa" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40309 } }
I20260806 09:15:35.599341 28932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:35.599458 28932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669ea0ba673040508846c6bfa94092aa, State: Initialized, Role: FOLLOWER
I20260806 09:15:35.600029 28932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669ea0ba673040508846c6bfa94092aa" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40309 } }
I20260806 09:15:35.600286 28932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:35.600428 28932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:35.600620 28932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:35.602785 28932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669ea0ba673040508846c6bfa94092aa" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40309 } }
I20260806 09:15:35.603349 28932 leader_election.cc:302] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 669ea0ba673040508846c6bfa94092aa; no voters:
I20260806 09:15:35.603992 28932 leader_election.cc:288] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:15:35.604171 28940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:35.604966 28940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [term 1 LEADER]: Becoming Leader. State: Replica: 669ea0ba673040508846c6bfa94092aa, State: Running, Role: LEADER
I20260806 09:15:35.605554 28940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669ea0ba673040508846c6bfa94092aa" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40309 } }
I20260806 09:15:35.606074 28932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:35.608439 28941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "669ea0ba673040508846c6bfa94092aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669ea0ba673040508846c6bfa94092aa" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40309 } } }
I20260806 09:15:35.608829 28941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:35.608959 28942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 669ea0ba673040508846c6bfa94092aa. Latest consensus state: current_term: 1 leader_uuid: "669ea0ba673040508846c6bfa94092aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669ea0ba673040508846c6bfa94092aa" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40309 } } }
I20260806 09:15:35.609419 28942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:35.611783 28948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:35.617545 28948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:35.626211 28948 catalog_manager.cc:1347] Generated new cluster ID: 7ed5927bf0674c2196ebe92f57b22ed0
I20260806 09:15:35.626324 28948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:35.662254 28948 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:15:35.664113 28948 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:35.679946 28948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa: Generated new TSK 0
I20260806 09:15:35.680974 28948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:15:35.691802 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40309
--builtin_ntp_servers=127.3.69.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:15:36.041543 28964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:36.041875 28964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:36.042047 28964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:36.049546 28964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:36.049690 28964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:36.049886 28964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:15:36.059051 28964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:36.060458 28964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:36.062314 28964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:36.075812 28972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:36.075912 28974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:36.075790 28976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:36.076884 28964 server_base.cc:1034] running on GCE node
I20260806 09:15:36.077437 28964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:36.078464 28964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:36.079756 28964 hybrid_clock.cc:648] HybridClock initialized: now 1786007736079718 us; error 31 us; skew 500 ppm
I20260806 09:15:36.080205 28964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:36.082579 28964 webserver.cc:460] Webserver started at http://127.3.69.1:45669/ using document root <none> and password file <none>
I20260806 09:15:36.083359 28964 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:36.083489 28964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:36.083837 28964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:36.086700 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "85968926eb9c4576b5f897db93b5ef3f"
format_stamp: "Formatted at 2026-08-06 09:15:36 on dist-test-slave-w4v5"
server_key: "4e3f4bc12dc2da103793fde27adf8e51"
server_key_iv: "958baf477d98ce54d6f853810fd4384c"
server_key_version: "encryptionkey@0"
I20260806 09:15:36.087661 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "85968926eb9c4576b5f897db93b5ef3f"
format_stamp: "Formatted at 2026-08-06 09:15:36 on dist-test-slave-w4v5"
server_key: "4e3f4bc12dc2da103793fde27adf8e51"
server_key_iv: "958baf477d98ce54d6f853810fd4384c"
server_key_version: "encryptionkey@0"
I20260806 09:15:36.095745 28964 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 09:15:36.100446 28991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:36.102017 28964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 09:15:36.102200 28964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "85968926eb9c4576b5f897db93b5ef3f"
format_stamp: "Formatted at 2026-08-06 09:15:36 on dist-test-slave-w4v5"
server_key: "4e3f4bc12dc2da103793fde27adf8e51"
server_key_iv: "958baf477d98ce54d6f853810fd4384c"
server_key_version: "encryptionkey@0"
I20260806 09:15:36.102442 28964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:36.149701 28964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:36.150743 28964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:36.151062 28964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:36.152141 28964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:36.153692 28964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:36.153828 28964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:36.153965 28964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:36.154064 28964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:36.187806 28964 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:37467
I20260806 09:15:36.187901 29156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:37467 every 8 connection(s)
I20260806 09:15:36.189220 28964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:15:36.198472 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 28964
I20260806 09:15:36.198654 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:15:36.199184 3348 external_mini_cluster.cc:1428] Setting key 641561eb07e8f03a1db9d7c850f5a47b
I20260806 09:15:36.202054 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40309
--builtin_ntp_servers=127.3.69.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:15:36.202302 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:36.191883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:48023 (local address 127.3.69.62:40309)
0806 09:15:36.192288 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:36.192293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:36.193613 (+ 1320us) server_negotiation.cc:407] Connection header received
0806 09:15:36.195523 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:36.195529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:36.195626 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:36.195752 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:36.196812 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:36.197527 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:36.199271 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:36.199720 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:36.199865 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:36.201376 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:36.201381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:36.201384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:36.201400 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:36.201440 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:36.201452 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:36.201548 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:36.201814 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 09:15:36.201919 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":205,"threads_started":1}
I20260806 09:15:36.203361 29161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:36.192486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40309 (local address 127.3.69.1:48023)
0806 09:15:36.193406 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:36.193442 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:36.195249 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:36.195975 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:36.195983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:36.196067 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:36.196635 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:36.196651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:36.197688 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:36.197697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:36.199133 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:36.199146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:36.199956 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:36.199964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:36.200198 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:36.201012 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:36.201015 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:36.201078 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:36.201162 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:36.201560 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:36.201573 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:36.201587 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 09:15:36.201800 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 09:15:36.202204 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":303,"threads_started":1}
I20260806 09:15:36.204996 29157 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40309
I20260806 09:15:36.205302 29157 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:36.206080 29157 heartbeater.cc:511] Master 127.3.69.62:40309 requested a full tablet report, sending...
I20260806 09:15:36.208520 28876 ts_manager.cc:194] Registered new tserver with Master: 85968926eb9c4576b5f897db93b5ef3f (127.3.69.1:37467)
I20260806 09:15:36.210667 28876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.1:48023
W20260806 09:15:36.557691 29163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:36.557981 29163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:36.558073 29163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:36.565459 29163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:36.565574 29163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:36.565750 29163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:15:36.575565 29163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:36.576917 29163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:36.578670 29163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:36.592154 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:36.592252 29172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:36.592204 29176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:36.593299 29163 server_base.cc:1034] running on GCE node
I20260806 09:15:36.593961 29163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:36.594973 29163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:36.596241 29163 hybrid_clock.cc:648] HybridClock initialized: now 1786007736596185 us; error 45 us; skew 500 ppm
I20260806 09:15:36.596606 29163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:36.599148 29163 webserver.cc:460] Webserver started at http://127.3.69.2:41371/ using document root <none> and password file <none>
I20260806 09:15:36.599969 29163 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:36.600108 29163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:36.600450 29163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:36.603617 29163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "d5ffbc0c7e114506aa73e2f6a4142865"
format_stamp: "Formatted at 2026-08-06 09:15:36 on dist-test-slave-w4v5"
server_key: "d4e4942ae26a18f7d0e65aae786246ff"
server_key_iv: "d7de4035be881702a04117b7aa16bc1a"
server_key_version: "encryptionkey@0"
I20260806 09:15:36.604410 29163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "d5ffbc0c7e114506aa73e2f6a4142865"
format_stamp: "Formatted at 2026-08-06 09:15:36 on dist-test-slave-w4v5"
server_key: "d4e4942ae26a18f7d0e65aae786246ff"
server_key_iv: "d7de4035be881702a04117b7aa16bc1a"
server_key_version: "encryptionkey@0"
I20260806 09:15:36.609784 29163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:15:36.613613 29187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:36.614957 29163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:15:36.615185 29163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "d5ffbc0c7e114506aa73e2f6a4142865"
format_stamp: "Formatted at 2026-08-06 09:15:36 on dist-test-slave-w4v5"
server_key: "d4e4942ae26a18f7d0e65aae786246ff"
server_key_iv: "d7de4035be881702a04117b7aa16bc1a"
server_key_version: "encryptionkey@0"
I20260806 09:15:36.615430 29163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:36.650754 29163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:36.651810 29163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:36.652118 29163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:36.653167 29163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:36.654750 29163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:36.654874 29163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:36.655009 29163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:36.655113 29163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:36.689688 29163 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:45043
I20260806 09:15:36.689781 29360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:45043 every 8 connection(s)
I20260806 09:15:36.691035 29163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:15:36.698091 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 29163
I20260806 09:15:36.698316 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:15:36.698905 3348 external_mini_cluster.cc:1428] Setting key fecebe00c84032ddfacc708452486cd5
I20260806 09:15:36.702252 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40309
--builtin_ntp_servers=127.3.69.20:43097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:15:36.705971 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:36.693829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:47825 (local address 127.3.69.62:40309)
0806 09:15:36.694032 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:36.694037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:36.695565 (+ 1528us) server_negotiation.cc:407] Connection header received
0806 09:15:36.697950 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:36.697955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:36.698064 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:36.698185 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:36.699358 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:36.700394 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:36.702854 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:36.703338 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:36.703441 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:36.705175 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:36.705179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:36.705182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:36.705200 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:36.705234 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:36.705245 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:36.705318 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:36.705633 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 09:15:36.705728 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:15:36.707508 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:36.694383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40309 (local address 127.3.69.2:47825)
0806 09:15:36.695379 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:36.695415 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:36.697672 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:36.698420 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:36.698428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:36.698524 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:36.699186 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:36.699206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:36.700592 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:36.700596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:36.702700 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:36.702719 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:36.703576 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:36.703587 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:36.703940 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:36.704787 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:36.704793 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:36.704848 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:36.704962 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:36.705347 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:36.705361 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:36.705372 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:15:36.705570 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 09:15:36.706015 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":311,"threads_started":1}
I20260806 09:15:36.709101 29361 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40309
I20260806 09:15:36.709442 29361 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:36.710220 29361 heartbeater.cc:511] Master 127.3.69.62:40309 requested a full tablet report, sending...
I20260806 09:15:36.711853 28876 ts_manager.cc:194] Registered new tserver with Master: d5ffbc0c7e114506aa73e2f6a4142865 (127.3.69.2:45043)
I20260806 09:15:36.712708 28876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.2:47825
W20260806 09:15:37.057027 29371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:37.057397 29371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:37.057567 29371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:37.064851 29371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:37.064996 29371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:37.065186 29371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:15:37.074234 29371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:37.075572 29371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:37.077445 29371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:37.090919 29376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:37.090967 29377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:37.090993 29382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:37.091990 29371 server_base.cc:1034] running on GCE node
I20260806 09:15:37.092584 29371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:37.093569 29371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:37.094781 29371 hybrid_clock.cc:648] HybridClock initialized: now 1786007737094735 us; error 32 us; skew 500 ppm
I20260806 09:15:37.095173 29371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:37.097448 29371 webserver.cc:460] Webserver started at http://127.3.69.3:34149/ using document root <none> and password file <none>
I20260806 09:15:37.098133 29371 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:37.098224 29371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:37.098520 29371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:37.101418 29371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "61e479fa62664d009a95b1a878cec382"
format_stamp: "Formatted at 2026-08-06 09:15:37 on dist-test-slave-w4v5"
server_key: "2d850f1349ab5358ee127a5358267194"
server_key_iv: "129d1470e6bc7b48917069e4209b331f"
server_key_version: "encryptionkey@0"
I20260806 09:15:37.102173 29371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "61e479fa62664d009a95b1a878cec382"
format_stamp: "Formatted at 2026-08-06 09:15:37 on dist-test-slave-w4v5"
server_key: "2d850f1349ab5358ee127a5358267194"
server_key_iv: "129d1470e6bc7b48917069e4209b331f"
server_key_version: "encryptionkey@0"
I20260806 09:15:37.107470 29371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:15:37.111244 29395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:37.112643 29371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:15:37.112828 29371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "61e479fa62664d009a95b1a878cec382"
format_stamp: "Formatted at 2026-08-06 09:15:37 on dist-test-slave-w4v5"
server_key: "2d850f1349ab5358ee127a5358267194"
server_key_iv: "129d1470e6bc7b48917069e4209b331f"
server_key_version: "encryptionkey@0"
I20260806 09:15:37.113049 29371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:37.138927 29371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:37.139812 29371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:37.140110 29371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:37.141119 29371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:37.142623 29371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:37.142766 29371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:37.142894 29371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:37.142987 29371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:37.177769 29371 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:38337
I20260806 09:15:37.177856 29566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:38337 every 8 connection(s)
I20260806 09:15:37.179140 29371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:15:37.187237 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 29371
I20260806 09:15:37.187419 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:15:37.187973 3348 external_mini_cluster.cc:1428] Setting key 07af253963817972c4385079720c5bbe
I20260806 09:15:37.192404 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.181742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:47161 (local address 127.3.69.62:40309)
0806 09:15:37.181925 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.181930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.183296 (+ 1366us) server_negotiation.cc:407] Connection header received
0806 09:15:37.185067 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.185072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.185144 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.185234 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:37.186264 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.187035 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.189011 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.189629 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.189757 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.191494 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:37.191501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:37.191505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:37.191528 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:37.191567 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:37.191583 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:37.191694 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.191939 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.192035 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:15:37.193372 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.182145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40309 (local address 127.3.69.3:47161)
0806 09:15:37.183067 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.183178 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:37.184803 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:37.185428 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:37.185435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:37.185523 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:37.186110 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.186126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.187330 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.187335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.188854 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.188867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.189851 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.189858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.190147 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.191042 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:37.191045 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:37.191137 (+ 92us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:37.191243 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:37.191696 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:37.191711 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:37.191720 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:15:37.191910 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 09:15:37.192289 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":282,"threads_started":1}
I20260806 09:15:37.194636 29567 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40309
I20260806 09:15:37.194944 29567 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:37.195694 29567 heartbeater.cc:511] Master 127.3.69.62:40309 requested a full tablet report, sending...
I20260806 09:15:37.197067 28876 ts_manager.cc:194] Registered new tserver with Master: 61e479fa62664d009a95b1a878cec382 (127.3.69.3:38337)
I20260806 09:15:37.197883 28876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.3:47161
I20260806 09:15:37.203394 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:15:37.213500 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.207637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.3.69.62:40309)
0806 09:15:37.207832 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.207836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.208340 (+ 504us) server_negotiation.cc:407] Connection header received
0806 09:15:37.208507 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.208511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.208573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.208661 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:37.209160 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.209827 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.211212 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.211718 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.211832 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.212870 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:37.212881 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:37.212889 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:37.212906 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:37.212939 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:37.212952 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:37.213040 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.213171 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.213243 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 09:15:37.213977 29157 heartbeater.cc:503] Master 127.3.69.62:40309 was elected leader, sending a full tablet report...
I20260806 09:15:37.217432 28876 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:15:37.224848 28876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:15:37.249483 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.239191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:45043 (local address 127.0.0.1:55842)
0806 09:15:37.239694 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.239711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:37.239814 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:37.241204 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:37.241212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:37.241241 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:37.241749 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.241768 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.244171 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.244175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.246544 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.246557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.248476 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.248483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.248694 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.248756 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 09:15:37.248874 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 09:15:37.249061 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":133,"thread_start_us":716,"threads_started":1}
I20260806 09:15:37.250124 29596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.237342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:38337 (local address 127.0.0.1:39280)
0806 09:15:37.238262 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.238298 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:37.238480 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:37.243055 (+ 4575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:37.243059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:37.243105 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:37.243501 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.243511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.245474 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.245482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.247513 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.247525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.249121 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.249128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.249309 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.249339 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 09:15:37.249427 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 09:15:37.249542 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":694,"threads_started":1}
I20260806 09:15:37.250171 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.237800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59784)
0806 09:15:37.239108 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.239126 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:37.239231 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:37.241456 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:37.241461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:37.241482 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:37.241922 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.241935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.244114 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.244119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.246820 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.246835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.248665 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.248671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.248873 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.248906 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:15:37.248976 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 09:15:37.249433 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"mutex_wait_us":154,"thread_start_us":980,"threads_started":1}
I20260806 09:15:37.250337 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.239737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.3.69.2:45043)
0806 09:15:37.240507 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.240517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.240551 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:15:37.240668 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.240677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.240947 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.242024 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:37.242274 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.243972 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.246747 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.248270 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.248417 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.248695 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.249771 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.249950 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":477,"threads_started":1}
I20260806 09:15:37.250957 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.237626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.3.69.3:38337)
0806 09:15:37.242599 (+ 4973us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.242605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.242631 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:15:37.242708 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.242713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.242848 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.243069 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:37.243697 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.245296 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.247695 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.248737 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.248883 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.249136 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.249700 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.250574 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4826,"thread_start_us":4658,"threads_started":1}
I20260806 09:15:37.251473 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.238343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.3.69.1:37467)
0806 09:15:37.240867 (+ 2524us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.240874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.240900 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:15:37.240992 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.240999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.241234 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.242090 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:37.242324 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.243964 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.247028 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.248429 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.250272 (+ 1843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.250852 (+ 580us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.250929 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.251128 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2358,"thread_start_us":2185,"threads_started":1}
I20260806 09:15:37.254482 29055 tablet_service.cc:1467] Processing CreateTablet for tablet 999830adeba34ee5aa4298f1c99f64e2 (DEFAULT_TABLE table=test-table [id=6b0b67d3e9f14fb99b696a4c6caf00e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:37.254455 29469 tablet_service.cc:1467] Processing CreateTablet for tablet 999830adeba34ee5aa4298f1c99f64e2 (DEFAULT_TABLE table=test-table [id=6b0b67d3e9f14fb99b696a4c6caf00e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:37.254455 29262 tablet_service.cc:1467] Processing CreateTablet for tablet 999830adeba34ee5aa4298f1c99f64e2 (DEFAULT_TABLE table=test-table [id=6b0b67d3e9f14fb99b696a4c6caf00e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:37.255921 29055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999830adeba34ee5aa4298f1c99f64e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:37.255949 29262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999830adeba34ee5aa4298f1c99f64e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:37.256217 29469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999830adeba34ee5aa4298f1c99f64e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:37.267215 29606 tablet_bootstrap.cc:492] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: Bootstrap starting.
I20260806 09:15:37.270661 29607 tablet_bootstrap.cc:492] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Bootstrap starting.
I20260806 09:15:37.271524 29609 tablet_bootstrap.cc:492] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Bootstrap starting.
I20260806 09:15:37.271596 29606 tablet_bootstrap.cc:654] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:37.272964 29606 log.cc:824] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:37.275393 29609 tablet_bootstrap.cc:654] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:37.275789 29606 tablet_bootstrap.cc:492] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: No bootstrap required, opened a new log
I20260806 09:15:37.276070 29606 ts_tablet_manager.cc:1397] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 09:15:37.276751 29609 log.cc:824] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:37.277305 29607 tablet_bootstrap.cc:654] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:37.278858 29607 log.cc:824] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:37.281821 29607 tablet_bootstrap.cc:492] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: No bootstrap required, opened a new log
I20260806 09:15:37.282106 29607 ts_tablet_manager.cc:1397] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260806 09:15:37.282272 29609 tablet_bootstrap.cc:492] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: No bootstrap required, opened a new log
I20260806 09:15:37.282545 29609 ts_tablet_manager.cc:1397] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260806 09:15:37.286002 29606 raft_consensus.cc:348] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.286477 29606 raft_consensus.cc:374] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:37.286592 29606 raft_consensus.cc:729] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85968926eb9c4576b5f897db93b5ef3f, State: Initialized, Role: FOLLOWER
I20260806 09:15:37.287256 29606 consensus_queue.cc:260] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.290076 29606 ts_tablet_manager.cc:1428] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 09:15:37.291175 29607 raft_consensus.cc:348] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.291549 29607 raft_consensus.cc:374] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:37.291677 29607 raft_consensus.cc:729] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e479fa62664d009a95b1a878cec382, State: Initialized, Role: FOLLOWER
I20260806 09:15:37.292256 29607 consensus_queue.cc:260] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.293263 29609 raft_consensus.cc:348] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.293802 29609 raft_consensus.cc:374] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:37.293915 29609 raft_consensus.cc:729] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ffbc0c7e114506aa73e2f6a4142865, State: Initialized, Role: FOLLOWER
I20260806 09:15:37.293967 29607 ts_tablet_manager.cc:1428] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 09:15:37.294590 29567 heartbeater.cc:503] Master 127.3.69.62:40309 was elected leader, sending a full tablet report...
I20260806 09:15:37.294497 29609 consensus_queue.cc:260] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.295998 29361 heartbeater.cc:503] Master 127.3.69.62:40309 was elected leader, sending a full tablet report...
I20260806 09:15:37.296226 29609 ts_tablet_manager.cc:1428] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260806 09:15:37.370174 29614 raft_consensus.cc:482] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:15:37.370543 29614 raft_consensus.cc:504] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.372435 29614 leader_election.cc:288] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85968926eb9c4576b5f897db93b5ef3f (127.3.69.1:37467), d5ffbc0c7e114506aa73e2f6a4142865 (127.3.69.2:45043)
I20260806 09:15:37.379302 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.372801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.3.69.3:46237)
0806 09:15:37.372989 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.373008 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:37.373162 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:37.373617 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:37.373648 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:37.373684 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:37.373942 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.373955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.375324 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.375328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.377252 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.377268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.378617 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.378622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.378746 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.378771 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:15:37.378855 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 09:15:37.378951 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 09:15:37.379341 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.372996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:46237 (local address 127.3.69.1:37467)
0806 09:15:37.373239 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.373245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.373266 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:15:37.373345 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.373350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.373440 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.373533 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:37.374145 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.375139 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.377420 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.378415 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.378546 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.378646 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.378910 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.379044 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:15:37.380020 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.373427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:47595 (local address 127.3.69.2:45043)
0806 09:15:37.373684 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.373694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.374314 (+ 620us) server_negotiation.cc:407] Connection header received
0806 09:15:37.374401 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.374408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.374509 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.374612 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:37.375138 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.375869 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.378020 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.379142 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.379264 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.379369 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.379723 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.379836 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:15:37.380133 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.373280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:45043 (local address 127.3.69.3:47595)
0806 09:15:37.374206 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.374219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:37.374311 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:37.374633 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:37.374636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:37.374655 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:37.374944 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.374950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.376061 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.376067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.377832 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:37.377844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.379323 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.379332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:37.379535 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.379567 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:15:37.379663 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 09:15:37.379813 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":534,"threads_started":1}
I20260806 09:15:37.380604 29088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999830adeba34ee5aa4298f1c99f64e2" candidate_uuid: "61e479fa62664d009a95b1a878cec382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85968926eb9c4576b5f897db93b5ef3f" is_pre_election: true
I20260806 09:15:37.381116 29088 raft_consensus.cc:2454] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e479fa62664d009a95b1a878cec382 in term 0.
I20260806 09:15:37.381168 29296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999830adeba34ee5aa4298f1c99f64e2" candidate_uuid: "61e479fa62664d009a95b1a878cec382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" is_pre_election: true
I20260806 09:15:37.381719 29296 raft_consensus.cc:2454] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e479fa62664d009a95b1a878cec382 in term 0.
I20260806 09:15:37.381830 29400 leader_election.cc:302] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e479fa62664d009a95b1a878cec382, 85968926eb9c4576b5f897db93b5ef3f; no voters:
I20260806 09:15:37.382272 29614 raft_consensus.cc:2781] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:15:37.382421 29614 raft_consensus.cc:482] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:15:37.382546 29614 raft_consensus.cc:3037] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:37.384867 29614 raft_consensus.cc:504] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.385464 29614 leader_election.cc:288] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [CANDIDATE]: Term 1 election: Requested vote from peers 85968926eb9c4576b5f897db93b5ef3f (127.3.69.1:37467), d5ffbc0c7e114506aa73e2f6a4142865 (127.3.69.2:45043)
I20260806 09:15:37.385902 29088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999830adeba34ee5aa4298f1c99f64e2" candidate_uuid: "61e479fa62664d009a95b1a878cec382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85968926eb9c4576b5f897db93b5ef3f"
I20260806 09:15:37.386067 29088 raft_consensus.cc:3037] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:37.386066 29296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999830adeba34ee5aa4298f1c99f64e2" candidate_uuid: "61e479fa62664d009a95b1a878cec382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ffbc0c7e114506aa73e2f6a4142865"
I20260806 09:15:37.386288 29296 raft_consensus.cc:3037] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:37.388579 29088 raft_consensus.cc:2454] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e479fa62664d009a95b1a878cec382 in term 1.
I20260806 09:15:37.388844 29296 raft_consensus.cc:2454] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e479fa62664d009a95b1a878cec382 in term 1.
I20260806 09:15:37.389011 29400 leader_election.cc:302] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e479fa62664d009a95b1a878cec382, 85968926eb9c4576b5f897db93b5ef3f; no voters:
I20260806 09:15:37.389331 29614 raft_consensus.cc:2781] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:37.389897 29614 raft_consensus.cc:686] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 1 LEADER]: Becoming Leader. State: Replica: 61e479fa62664d009a95b1a878cec382, State: Running, Role: LEADER
I20260806 09:15:37.390435 29614 consensus_queue.cc:237] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } }
I20260806 09:15:37.394868 28876 catalog_manager.cc:5495] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61e479fa62664d009a95b1a878cec382 (127.3.69.3). New cstate: current_term: 1 leader_uuid: "61e479fa62664d009a95b1a878cec382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61e479fa62664d009a95b1a878cec382" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 } health_report { overall_health: UNKNOWN } } }
W20260806 09:15:37.430824 29568 tablet.cc:2369] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:37.440049 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.434870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.3.69.3:38337)
0806 09:15:37.435059 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.435064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.435109 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:15:37.435344 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.435348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.435426 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.435556 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:15:37.435898 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.436621 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.438042 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.438449 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.438543 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.439052 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:15:37.439562 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:15:37.439678 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.439800 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.439861 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 09:15:37.441866 29158 tablet.cc:2369] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:15:37.443164 29366 tablet.cc:2369] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:37.448276 29088 raft_consensus.cc:1261] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 1 FOLLOWER]: Refusing update from remote peer 61e479fa62664d009a95b1a878cec382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:37.448289 29296 raft_consensus.cc:1261] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 1 FOLLOWER]: Refusing update from remote peer 61e479fa62664d009a95b1a878cec382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:37.449224 29617 consensus_queue.cc:1035] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85968926eb9c4576b5f897db93b5ef3f" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:37.449575 29614 consensus_queue.cc:1035] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ffbc0c7e114506aa73e2f6a4142865" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:37.464210 29625 mvcc.cc:205] Tried to move back new op lower bound from 7315487692577480704 to 7315487692355981312. Current Snapshot: MvccSnapshot[applied={T|T < 7315487692577480704}]
I20260806 09:15:37.474696 28876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:37.474938 28876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:37.478502 28876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 669ea0ba673040508846c6bfa94092aa: Sending DeleteTablet for 3 replicas of tablet 999830adeba34ee5aa4298f1c99f64e2
I20260806 09:15:37.479732 29469 tablet_service.cc:1514] Processing DeleteTablet for tablet 999830adeba34ee5aa4298f1c99f64e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:37 UTC) from {username='slave'} at 127.0.0.1:39280
I20260806 09:15:37.479772 29055 tablet_service.cc:1514] Processing DeleteTablet for tablet 999830adeba34ee5aa4298f1c99f64e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:37 UTC) from {username='slave'} at 127.0.0.1:59784
I20260806 09:15:37.479862 29262 tablet_service.cc:1514] Processing DeleteTablet for tablet 999830adeba34ee5aa4298f1c99f64e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:37 UTC) from {username='slave'} at 127.0.0.1:55842
I20260806 09:15:37.482087 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 28964
I20260806 09:15:37.495182 29641 tablet_replica.cc:330] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: stopping tablet replica
I20260806 09:15:37.495862 29641 raft_consensus.cc:2229] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:37.496515 29641 raft_consensus.cc:2258] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:37.499231 29641 ts_tablet_manager.cc:1905] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:37.508221 29641 ts_tablet_manager.cc:1918] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:37.508452 29641 log.cc:1196] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/999830adeba34ee5aa4298f1c99f64e2
I20260806 09:15:37.509087 29641 ts_tablet_manager.cc:1939] T 999830adeba34ee5aa4298f1c99f64e2 P d5ffbc0c7e114506aa73e2f6a4142865: Deleting consensus metadata
I20260806 09:15:37.511266 28855 catalog_manager.cc:4841] TS d5ffbc0c7e114506aa73e2f6a4142865 (127.3.69.2:45043): tablet 999830adeba34ee5aa4298f1c99f64e2 (table test-table [id=6b0b67d3e9f14fb99b696a4c6caf00e0]) successfully deleted
I20260806 09:15:37.513398 29639 tablet_replica.cc:330] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: stopping tablet replica
I20260806 09:15:37.514012 29639 raft_consensus.cc:2229] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:15:37.514987 29639 raft_consensus.cc:2258] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:37.515826 29640 tablet_replica.cc:330] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: stopping tablet replica
I20260806 09:15:37.516218 29640 raft_consensus.cc:2229] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:37.516835 29640 raft_consensus.cc:2258] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:37.518190 29639 ts_tablet_manager.cc:1905] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:37.519037 29640 ts_tablet_manager.cc:1905] T 999830adeba34ee5aa4298f1c99f64e2 P 85968926eb9c4576b5f897db93b5ef3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:37.520363 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.483394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.3.69.1:37467)
0806 09:15:37.483624 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.483634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.508092 (+ 24458us) server_negotiation.cc:407] Connection header received
0806 09:15:37.508503 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.508511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.508694 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.508792 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:37.509750 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.511003 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.512862 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.513566 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.513696 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.518497 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:37.518508 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:37.518523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:37.519326 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:37.519398 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:37.519423 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:37.519514 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.519857 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.519978 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:15:37.521736 29147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:37.526434 29639 ts_tablet_manager.cc:1918] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:37.526630 29639 log.cc:1196] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/999830adeba34ee5aa4298f1c99f64e2
I20260806 09:15:37.527220 29639 ts_tablet_manager.cc:1939] T 999830adeba34ee5aa4298f1c99f64e2 P 61e479fa62664d009a95b1a878cec382: Deleting consensus metadata
I20260806 09:15:37.528862 28853 catalog_manager.cc:4841] TS 61e479fa62664d009a95b1a878cec382 (127.3.69.3:38337): tablet 999830adeba34ee5aa4298f1c99f64e2 (table test-table [id=6b0b67d3e9f14fb99b696a4c6caf00e0]) successfully deleted
I20260806 09:15:37.640807 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 28964
I20260806 09:15:37.669684 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 29163
W20260806 09:15:37.672580 28855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.69.1:37467 (error 108)
I20260806 09:15:37.675369 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.674824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59812)
0806 09:15:37.674987 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.675138 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 09:15:37.675768 28855 catalog_manager.cc:4584] TS 85968926eb9c4576b5f897db93b5ef3f (127.3.69.1:37467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 999830adeba34ee5aa4298f1c99f64e2: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
I20260806 09:15:37.681565 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.673910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.3.69.2:45043)
0806 09:15:37.674119 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.674127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.674413 (+ 286us) server_negotiation.cc:407] Connection header received
0806 09:15:37.674494 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.674499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.674641 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.674755 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:37.675538 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.676720 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.678029 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.678675 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.678804 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.679979 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:37.679990 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:37.680008 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:37.680800 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:37.680859 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:37.680884 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:37.680997 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.681092 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.681203 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:15:37.682314 29355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:37.729777 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.729182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59828)
0806 09:15:37.729440 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.729560 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 09:15:37.731106 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.730647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59832)
0806 09:15:37.730828 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.730927 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 09:15:37.774121 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 29163
I20260806 09:15:37.791877 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.791273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59840)
0806 09:15:37.791538 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.791666 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 09:15:37.793217 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.792721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59850)
0806 09:15:37.792910 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.793050 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 09:15:37.803156 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 29371
I20260806 09:15:37.811252 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.804046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.3.69.3:38337)
0806 09:15:37.804226 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:37.804230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:37.804295 (+ 65us) server_negotiation.cc:407] Connection header received
0806 09:15:37.804508 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:37.804512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:37.804592 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:37.804683 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:37.805239 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.806026 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.808045 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:37.808482 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:37.808580 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:37.809786 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:37.809795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:37.809810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:37.810392 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:37.810442 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:37.810460 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:37.810554 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:37.810749 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 09:15:37.810837 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:15:37.812063 29559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:37.874329 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.873766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59864)
0806 09:15:37.874030 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.874137 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 09:15:37.875576 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:37.875157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59870)
0806 09:15:37.875366 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:37.875441 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 09:15:37.935622 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 29371
I20260806 09:15:37.963856 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 28832
I20260806 09:15:37.965020 28923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:38.077636 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 28832
I20260806 09:15:38.077709 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:38.077178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37467 (local address 127.0.0.1:59872)
0806 09:15:38.077380 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:38.077479 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:37467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
2026-08-06T09:15:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3069 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:15:38 dist-test-slave-w4v5 krb5kdc[29661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:15:38 dist-test-slave-w4v5 krb5kdc[29661](info): set up 2 sockets
Aug 06 09:15:38 dist-test-slave-w4v5 krb5kdc[29661](info): commencing operation
krb5kdc: starting...
W20260806 09:15:39.399158 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.268s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:15:39 dist-test-slave-w4v5 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:15:39.715467 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:35703
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:35703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:40.063849 29676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:40.064136 29676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:40.064287 29676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:40.071476 29676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:40.071576 29676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:40.071686 29676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:40.071808 29676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:40.071871 29676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:40.081591 29676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:35703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:35703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:40.082849 29676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:40.084173 29676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:40.096473 29686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:40.096779 29682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:40.096918 29683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:40.098012 29676 server_base.cc:1034] running on GCE node
I20260806 09:15:40.098840 29676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:40.100311 29676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:40.101584 29676 hybrid_clock.cc:648] HybridClock initialized: now 1786007740101511 us; error 58 us; skew 500 ppm
I20260806 09:15:40.101949 29676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:40 dist-test-slave-w4v5 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:40.105194 29676 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:15:40.106812 29676 webserver.cc:460] Webserver started at http://127.3.69.62:39907/ using document root <none> and password file <none>
I20260806 09:15:40.107594 29676 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:40.107746 29676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:40.108062 29676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:40.111052 29676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "226450469a8d4c1090f29978d4c03ede"
format_stamp: "Formatted at 2026-08-06 09:15:40 on dist-test-slave-w4v5"
server_key: "e8e32ae825b8fe25098dd0024bc60af3"
server_key_iv: "39b67f9abe1cd915142fbe3c2bc59554"
server_key_version: "encryptionkey@0"
I20260806 09:15:40.111928 29676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "226450469a8d4c1090f29978d4c03ede"
format_stamp: "Formatted at 2026-08-06 09:15:40 on dist-test-slave-w4v5"
server_key: "e8e32ae825b8fe25098dd0024bc60af3"
server_key_iv: "39b67f9abe1cd915142fbe3c2bc59554"
server_key_version: "encryptionkey@0"
I20260806 09:15:40.117183 29676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:15:40.120733 29694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:40.121969 29676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:15:40.122165 29676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "226450469a8d4c1090f29978d4c03ede"
format_stamp: "Formatted at 2026-08-06 09:15:40 on dist-test-slave-w4v5"
server_key: "e8e32ae825b8fe25098dd0024bc60af3"
server_key_iv: "39b67f9abe1cd915142fbe3c2bc59554"
server_key_version: "encryptionkey@0"
I20260806 09:15:40.122378 29676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:40.139482 29676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:40.143421 29676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:40.143711 29676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:40.163065 29676 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:35703
I20260806 09:15:40.163162 29775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:35703 every 8 connection(s)
I20260806 09:15:40.164445 29676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:40.167928 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 29676
I20260806 09:15:40.168128 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:15:40.168241 29779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:40.168646 3348 external_mini_cluster.cc:1428] Setting key c2c900c20f92d40f23a7fa2861ec20d9
I20260806 09:15:40.182854 29779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede: Bootstrap starting.
Aug 06 09:15:40 dist-test-slave-w4v5 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:40.186215 29779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:40.187490 29779 log.cc:824] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:40.190331 29779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede: No bootstrap required, opened a new log
I20260806 09:15:40.192662 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:40.170416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.3.69.62:35703)
0806 09:15:40.171338 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:40.171349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:40.171391 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:15:40.173044 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:40.173067 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:40.173427 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:40.173738 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:40.175659 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:40.177146 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:40.179163 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:40.179919 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:40.180086 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:40.185073 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:40.185081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:40.185095 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:40.185149 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:40.188475 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:40.189228 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:40.189236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:40.189242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:40.189408 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:40.189841 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:40.189846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:40.189849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:40.190364 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:40.190680 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:40.191267 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 09:15:40.191695 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":408,"threads_started":1}
I20260806 09:15:40.198307 29779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226450469a8d4c1090f29978d4c03ede" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 35703 } }
I20260806 09:15:40.198688 29779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:40.198797 29779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226450469a8d4c1090f29978d4c03ede, State: Initialized, Role: FOLLOWER
I20260806 09:15:40.199396 29779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226450469a8d4c1090f29978d4c03ede" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 35703 } }
I20260806 09:15:40.199646 29779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:40.199790 29779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:40.199975 29779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:40.202307 29779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226450469a8d4c1090f29978d4c03ede" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 35703 } }
I20260806 09:15:40.202797 29779 leader_election.cc:302] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 226450469a8d4c1090f29978d4c03ede; no voters:
I20260806 09:15:40.203475 29779 leader_election.cc:288] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:15:40.203608 29787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:40.205101 29787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [term 1 LEADER]: Becoming Leader. State: Replica: 226450469a8d4c1090f29978d4c03ede, State: Running, Role: LEADER
I20260806 09:15:40.205693 29779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:40.205583 29787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226450469a8d4c1090f29978d4c03ede" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 35703 } }
I20260806 09:15:40.208110 29789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226450469a8d4c1090f29978d4c03ede. Latest consensus state: current_term: 1 leader_uuid: "226450469a8d4c1090f29978d4c03ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226450469a8d4c1090f29978d4c03ede" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 35703 } } }
I20260806 09:15:40.208525 29789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:40.209085 29788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "226450469a8d4c1090f29978d4c03ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226450469a8d4c1090f29978d4c03ede" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 35703 } } }
I20260806 09:15:40.209396 29788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:40.210510 29795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:40.216041 29795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:40.225701 29795 catalog_manager.cc:1347] Generated new cluster ID: eb1bfb8ea1014cdfab84cc8c94b58a0b
I20260806 09:15:40.225867 29795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:40.240975 29795 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:15:40.242477 29795 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:40.254952 29795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede: Generated new TSK 0
I20260806 09:15:40.256096 29795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:15:40.342891 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:35703
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:40.697069 29814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:40.697513 29814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:40.697634 29814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:40.705073 29814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:40.705186 29814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:40.705302 29814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:15:40.715742 29814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:35703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:40.716990 29814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:40.718284 29814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:40.731714 29820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:40.731673 29823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:40.731879 29819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:40.732649 29814 server_base.cc:1034] running on GCE node
I20260806 09:15:40.733245 29814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:40.734113 29814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:40.735311 29814 hybrid_clock.cc:648] HybridClock initialized: now 1786007740735273 us; error 25 us; skew 500 ppm
I20260806 09:15:40.735579 29814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:40 dist-test-slave-w4v5 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:40.738686 29814 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:15:40.740166 29814 webserver.cc:460] Webserver started at http://127.3.69.1:44389/ using document root <none> and password file <none>
I20260806 09:15:40.740792 29814 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:40.740893 29814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:40.741151 29814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:40.744016 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "8f03cb875e8c47ae8bcbfa3f4795b726"
format_stamp: "Formatted at 2026-08-06 09:15:40 on dist-test-slave-w4v5"
server_key: "37bb1c07dc4025b0cbb761de26a8b73a"
server_key_iv: "ee90afe1b8a8b43097c4388a1f6bd137"
server_key_version: "encryptionkey@0"
I20260806 09:15:40.744745 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "8f03cb875e8c47ae8bcbfa3f4795b726"
format_stamp: "Formatted at 2026-08-06 09:15:40 on dist-test-slave-w4v5"
server_key: "37bb1c07dc4025b0cbb761de26a8b73a"
server_key_iv: "ee90afe1b8a8b43097c4388a1f6bd137"
server_key_version: "encryptionkey@0"
I20260806 09:15:40.749665 29814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:15:40.752976 29839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:40.754252 29814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:15:40.754422 29814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "8f03cb875e8c47ae8bcbfa3f4795b726"
format_stamp: "Formatted at 2026-08-06 09:15:40 on dist-test-slave-w4v5"
server_key: "37bb1c07dc4025b0cbb761de26a8b73a"
server_key_iv: "ee90afe1b8a8b43097c4388a1f6bd137"
server_key_version: "encryptionkey@0"
I20260806 09:15:40.754597 29814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:40.770079 29814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:40.773895 29814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:40.774184 29814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:40.775179 29814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:40.776503 29814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:40.776580 29814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:40.776749 29814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:40.776810 29814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:40.812700 29814 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:37365
I20260806 09:15:40.812788 30015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:37365 every 8 connection(s)
I20260806 09:15:40.814002 29814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:15:40.819284 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 29814
I20260806 09:15:40.819484 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:15:40.820118 3348 external_mini_cluster.cc:1428] Setting key 1d91362df66a0f9ae19d4bf40c829d10
Aug 06 09:15:40 dist-test-slave-w4v5 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:40.839988 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:40.816647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:53519 (local address 127.3.69.62:35703)
0806 09:15:40.817132 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:40.817138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:40.818244 (+ 1106us) server_negotiation.cc:407] Connection header received
0806 09:15:40.821015 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:40.821020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:40.821092 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:40.821187 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:40.823283 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:40.823997 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:40.826817 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:40.827333 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:40.827449 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:40.832295 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:40.832305 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:40.832314 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:40.832366 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:40.836353 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:40.837265 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:40.837274 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:40.837280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:40.837452 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:40.837964 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:40.837972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:40.837979 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:40.838337 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:40.838529 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:40.839329 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 09:15:40.839588 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260806 09:15:40.840965 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:40.817193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:35703 (local address 127.3.69.1:53519)
0806 09:15:40.818037 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:40.818095 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:40.820692 (+ 2597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:40.821463 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:40.821471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:40.822162 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:40.823098 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:40.823127 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:40.824171 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:40.824181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:40.826665 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:40.826684 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:40.827597 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:40.827607 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:40.828003 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:40.828807 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:40.828846 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:40.832030 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:40.836504 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:40.836515 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:40.836533 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:40.837114 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:40.837583 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:40.837590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:40.837596 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:40.837822 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:40.838493 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:40.838502 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:40.838860 (+ 358us) client_negotiation.cc:769] Sending connection context
0806 09:15:40.839196 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 09:15:40.839757 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":349,"threads_started":1}
I20260806 09:15:40.842818 30017 heartbeater.cc:347] Connected to a master server at 127.3.69.62:35703
I20260806 09:15:40.843230 30017 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:40.844039 30017 heartbeater.cc:511] Master 127.3.69.62:35703 requested a full tablet report, sending...
I20260806 09:15:40.846443 29721 ts_manager.cc:194] Registered new tserver with Master: 8f03cb875e8c47ae8bcbfa3f4795b726 (127.3.69.1:37365)
I20260806 09:15:40.848671 29721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:53519
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:15:40.894764 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:35703
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:41.239733 30023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:41.240031 30023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:41.240216 30023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:41.247964 30023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:41.248124 30023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:41.248324 30023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:15:41.258152 30023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:35703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:41.259560 30023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:41.260988 30023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:41.275153 30030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:41.275231 30033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:41.275203 30036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:41.276118 30023 server_base.cc:1034] running on GCE node
I20260806 09:15:41.276752 30023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:41.277750 30023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:41.278986 30023 hybrid_clock.cc:648] HybridClock initialized: now 1786007741278940 us; error 61 us; skew 500 ppm
I20260806 09:15:41.279340 30023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:41 dist-test-slave-w4v5 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:41.282593 30023 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:15:41.284139 30023 webserver.cc:460] Webserver started at http://127.3.69.2:34037/ using document root <none> and password file <none>
I20260806 09:15:41.284837 30023 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:41.284982 30023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:41.285303 30023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:41.288267 30023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "871b6e062d3148de9ce1dcecac8b55f8"
format_stamp: "Formatted at 2026-08-06 09:15:41 on dist-test-slave-w4v5"
server_key: "cdb7824607f96ed916a8710a5ec0c6f9"
server_key_iv: "e6b47758b1111636b86f6ac6f342ff0f"
server_key_version: "encryptionkey@0"
I20260806 09:15:41.289076 30023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "871b6e062d3148de9ce1dcecac8b55f8"
format_stamp: "Formatted at 2026-08-06 09:15:41 on dist-test-slave-w4v5"
server_key: "cdb7824607f96ed916a8710a5ec0c6f9"
server_key_iv: "e6b47758b1111636b86f6ac6f342ff0f"
server_key_version: "encryptionkey@0"
I20260806 09:15:41.294431 30023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:15:41.297895 30045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:41.299345 30023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 09:15:41.299526 30023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "871b6e062d3148de9ce1dcecac8b55f8"
format_stamp: "Formatted at 2026-08-06 09:15:41 on dist-test-slave-w4v5"
server_key: "cdb7824607f96ed916a8710a5ec0c6f9"
server_key_iv: "e6b47758b1111636b86f6ac6f342ff0f"
server_key_version: "encryptionkey@0"
I20260806 09:15:41.299731 30023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:41.319772 30023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:41.323863 30023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:41.324155 30023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:41.325134 30023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:41.326598 30023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:41.326718 30023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:41.326860 30023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:41.326937 30023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:41.361578 30023 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:44007
I20260806 09:15:41.361620 30234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:44007 every 8 connection(s)
I20260806 09:15:41.362880 30023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:15:41.367571 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 30023
I20260806 09:15:41.367774 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:15:41.368351 3348 external_mini_cluster.cc:1428] Setting key e79da86c2dd344f33c825b2074eaecd3
Aug 06 09:15:41 dist-test-slave-w4v5 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:41.385466 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.365407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:38703 (local address 127.3.69.62:35703)
0806 09:15:41.365914 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:41.365919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:41.366669 (+ 750us) server_negotiation.cc:407] Connection header received
0806 09:15:41.369298 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:41.369303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:41.369395 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:41.369519 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:41.371372 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.372093 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.374624 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.375144 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.375262 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:41.379597 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:41.379602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:41.379605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:41.379640 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:41.382348 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:41.383110 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:41.383121 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:41.383125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:41.383256 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:41.383725 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:41.383729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:41.383731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:41.383991 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:41.384152 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:41.384859 (+ 707us) server_negotiation.cc:299] Negotiation successful
0806 09:15:41.385084 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":304,"threads_started":1}
I20260806 09:15:41.386441 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.365789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:35703 (local address 127.3.69.2:38703)
0806 09:15:41.366494 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:41.366533 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:41.369015 (+ 2482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:41.369775 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:41.369783 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:41.370322 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:41.371182 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:41.371209 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.372254 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.372263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:41.374443 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:41.374462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.375388 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.375398 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:41.375726 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:41.376501 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:41.376544 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:41.379353 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:41.382500 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:41.382509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:41.382524 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:41.382953 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:41.383376 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:41.383381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:41.383385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:41.383582 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:41.384126 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:41.384134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:41.384473 (+ 339us) client_negotiation.cc:769] Sending connection context
0806 09:15:41.384741 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 09:15:41.385274 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":286,"threads_started":1}
I20260806 09:15:41.388072 30235 heartbeater.cc:347] Connected to a master server at 127.3.69.62:35703
I20260806 09:15:41.388388 30235 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:41.389128 30235 heartbeater.cc:511] Master 127.3.69.62:35703 requested a full tablet report, sending...
I20260806 09:15:41.390625 29721 ts_manager.cc:194] Registered new tserver with Master: 871b6e062d3148de9ce1dcecac8b55f8 (127.3.69.2:44007)
I20260806 09:15:41.391515 29721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:38703
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:15:41.442800 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:35703
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:41.799624 30242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:41.799954 30242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:41.800134 30242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:41.807650 30242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:41.807792 30242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:41.807976 30242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:15:41.820428 30242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:35703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:41.821722 30242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:41.823060 30242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:41.836989 30248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:41.837086 30249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:41.837134 30251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:41.838124 30242 server_base.cc:1034] running on GCE node
I20260806 09:15:41.838776 30242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:41.839852 30242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:41.841099 30242 hybrid_clock.cc:648] HybridClock initialized: now 1786007741841052 us; error 59 us; skew 500 ppm
I20260806 09:15:41.841471 30242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:41 dist-test-slave-w4v5 krb5kdc[29661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:41.844779 30242 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:15:41.846225 30242 webserver.cc:460] Webserver started at http://127.3.69.3:35261/ using document root <none> and password file <none>
I20260806 09:15:41.846930 30242 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:41.847112 30242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:41.847451 30242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:41.850349 30242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "eea7b189a4d24fb985f11924cec03611"
format_stamp: "Formatted at 2026-08-06 09:15:41 on dist-test-slave-w4v5"
server_key: "634481e4bb4c3b6288c1718e2b09ec56"
server_key_iv: "1e7ff4b8293884c458211c1de33a28b6"
server_key_version: "encryptionkey@0"
I20260806 09:15:41.851285 30242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "eea7b189a4d24fb985f11924cec03611"
format_stamp: "Formatted at 2026-08-06 09:15:41 on dist-test-slave-w4v5"
server_key: "634481e4bb4c3b6288c1718e2b09ec56"
server_key_iv: "1e7ff4b8293884c458211c1de33a28b6"
server_key_version: "encryptionkey@0"
I20260806 09:15:41.851972 30017 heartbeater.cc:503] Master 127.3.69.62:35703 was elected leader, sending a full tablet report...
I20260806 09:15:41.856899 30242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 09:15:41.860595 30265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:41.861860 30242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:15:41.862042 30242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "eea7b189a4d24fb985f11924cec03611"
format_stamp: "Formatted at 2026-08-06 09:15:41 on dist-test-slave-w4v5"
server_key: "634481e4bb4c3b6288c1718e2b09ec56"
server_key_iv: "1e7ff4b8293884c458211c1de33a28b6"
server_key_version: "encryptionkey@0"
I20260806 09:15:41.862255 30242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:41.878887 30242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:41.882921 30242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:41.883240 30242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:41.884284 30242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:41.885625 30242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:41.885738 30242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:41.885897 30242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:41.885972 30242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:41.921310 30242 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:38675
I20260806 09:15:41.921357 30443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:38675 every 8 connection(s)
I20260806 09:15:41.922712 30242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:15:41.924898 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 30242
I20260806 09:15:41.925110 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:15:41.925731 3348 external_mini_cluster.cc:1428] Setting key 496eabce91661148a2eb5ba40123c67c
Aug 06 09:15:41 dist-test-slave-w4v5 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:41.943759 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.926075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:57579 (local address 127.3.69.62:35703)
0806 09:15:41.926582 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:41.926590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:41.927501 (+ 911us) server_negotiation.cc:407] Connection header received
0806 09:15:41.929760 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:41.929764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:41.929843 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:41.929937 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:41.931533 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.932289 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.933814 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.934192 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.934283 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:41.937736 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:41.937742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:41.937745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:41.937776 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:41.940705 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:41.941659 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:41.941663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:41.941665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:41.941809 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:41.942211 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:41.942214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:41.942216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:41.942464 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:41.942583 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:41.943294 (+ 711us) server_negotiation.cc:299] Negotiation successful
0806 09:15:41.943467 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
I20260806 09:15:41.944792 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.926282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:35703 (local address 127.3.69.3:57579)
0806 09:15:41.927299 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:41.927376 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:41.929482 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:41.930166 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:41.930174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:41.930710 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:41.931380 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:41.931402 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.932452 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.932456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:41.933687 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:41.933699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.934379 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.934384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:41.934581 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:41.935312 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:41.935334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:41.937545 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:41.940857 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:41.940866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:41.940881 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:41.941496 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:41.941907 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:41.941910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:41.941912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:41.942106 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:41.942572 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:41.942578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:41.942880 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 09:15:41.943193 (+ 313us) client_negotiation.cc:240] Negotiation successful
0806 09:15:41.943580 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":369,"threads_started":1}
I20260806 09:15:41.946081 30445 heartbeater.cc:347] Connected to a master server at 127.3.69.62:35703
I20260806 09:15:41.946352 30445 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:41.947057 30445 heartbeater.cc:511] Master 127.3.69.62:35703 requested a full tablet report, sending...
I20260806 09:15:41.948550 29721 ts_manager.cc:194] Registered new tserver with Master: eea7b189a4d24fb985f11924cec03611 (127.3.69.3:38675)
I20260806 09:15:41.949389 29721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:57579
I20260806 09:15:41.953855 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:15:41.970736 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.957905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.3.69.62:35703)
0806 09:15:41.958068 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:41.958072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:41.958576 (+ 504us) server_negotiation.cc:407] Connection header received
0806 09:15:41.958693 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:41.958698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:41.958755 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:41.958835 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:41.960397 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.961142 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.962368 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:41.962866 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:41.962961 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:41.965667 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:41.965672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:41.965675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:41.965707 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:41.968316 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:41.969027 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:41.969031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:41.969033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:41.969146 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:41.969531 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:41.969534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:41.969536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:41.969823 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:41.970021 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:41.970337 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 09:15:41.970494 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:15:41.974637 29721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:15:41.981649 29721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:15:42.007997 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.994188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.3.69.1:37365)
0806 09:15:41.997515 (+ 3327us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:41.997527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:41.997564 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:15:41.997695 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:41.997707 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:41.998013 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:41.999633 (+ 1620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:41.999889 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.001526 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.004742 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.006292 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.006419 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.006657 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.007441 (+ 784us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.007625 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3112,"thread_start_us":2898,"threads_started":1}
I20260806 09:15:42.008005 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.994031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:37365 (local address 127.0.0.1:52708)
0806 09:15:41.995159 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:41.995230 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:41.995458 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:41.998768 (+ 3310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:41.998779 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:41.998826 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:41.999514 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:41.999531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.001703 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.001708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.004544 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.004558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.006815 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.006823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.007168 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.007228 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 09:15:42.007355 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 09:15:42.007544 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":54,"thread_start_us":594,"threads_started":1}
I20260806 09:15:42.009341 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.994224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:44007 (local address 127.0.0.1:37742)
0806 09:15:42.000013 (+ 5789us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:42.000035 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:42.000169 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:42.001065 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:42.001070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:42.001095 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:42.001526 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.001537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.003517 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.003524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.005889 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.005905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.008476 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.008485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.008676 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.008708 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:15:42.008818 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 09:15:42.008952 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5652,"mutex_wait_us":72,"thread_start_us":5196,"threads_started":1}
I20260806 09:15:42.009948 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.994692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.3.69.2:44007)
0806 09:15:41.996798 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:41.996807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:42.000293 (+ 3486us) server_negotiation.cc:407] Connection header received
0806 09:15:42.000399 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:42.000407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:42.000641 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:42.000819 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:42.001720 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.003356 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.006131 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.007672 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.007793 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.007989 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.008880 (+ 891us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.009053 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1932,"thread_start_us":1820,"threads_started":1}
I20260806 09:15:42.012234 29915 tablet_service.cc:1467] Processing CreateTablet for tablet 250bb071458543c28df51f71ade5e056 (DEFAULT_TABLE table=test-table [id=64b0d2165d094a63b7134854d78ffe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:42.013792 29915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250bb071458543c28df51f71ade5e056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:42.013629 30128 tablet_service.cc:1467] Processing CreateTablet for tablet 250bb071458543c28df51f71ade5e056 (DEFAULT_TABLE table=test-table [id=64b0d2165d094a63b7134854d78ffe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:42.015020 30128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250bb071458543c28df51f71ade5e056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:42.015542 30472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.996770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:38675 (local address 127.0.0.1:47206)
0806 09:15:42.006597 (+ 9827us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:42.006614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:42.006726 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:42.008016 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:42.008023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:42.008052 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:42.008461 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.008475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.010503 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.010507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.012604 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.012622 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.014720 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.014727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.014897 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.014934 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:15:42.015030 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 09:15:42.015181 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9699,"thread_start_us":9260,"threads_started":1}
I20260806 09:15:42.016330 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:41.996822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.3.69.3:38675)
0806 09:15:42.007003 (+ 10181us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:42.007012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:42.007053 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:15:42.007309 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:42.007315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:42.007549 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:42.007731 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:42.008683 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.010297 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.012800 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.014195 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.014333 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.014551 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.015856 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.016017 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9989,"thread_start_us":9039,"threads_started":1}
I20260806 09:15:42.019992 30339 tablet_service.cc:1467] Processing CreateTablet for tablet 250bb071458543c28df51f71ade5e056 (DEFAULT_TABLE table=test-table [id=64b0d2165d094a63b7134854d78ffe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:42.021514 30339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 250bb071458543c28df51f71ade5e056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:42.029893 30475 tablet_bootstrap.cc:492] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Bootstrap starting.
I20260806 09:15:42.034647 30475 tablet_bootstrap.cc:654] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:42.034768 30474 tablet_bootstrap.cc:492] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Bootstrap starting.
I20260806 09:15:42.036244 30475 log.cc:824] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:42.036365 30476 tablet_bootstrap.cc:492] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Bootstrap starting.
I20260806 09:15:42.038612 30474 tablet_bootstrap.cc:654] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:42.039733 30474 log.cc:824] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:42.039870 30476 tablet_bootstrap.cc:654] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:42.040164 30475 tablet_bootstrap.cc:492] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: No bootstrap required, opened a new log
I20260806 09:15:42.040493 30475 ts_tablet_manager.cc:1397] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260806 09:15:42.041178 30476 log.cc:824] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:42.049075 30476 tablet_bootstrap.cc:492] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: No bootstrap required, opened a new log
I20260806 09:15:42.049314 30474 tablet_bootstrap.cc:492] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: No bootstrap required, opened a new log
I20260806 09:15:42.049419 30476 ts_tablet_manager.cc:1397] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260806 09:15:42.049590 30474 ts_tablet_manager.cc:1397] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260806 09:15:42.052772 30475 raft_consensus.cc:348] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.053236 30475 raft_consensus.cc:374] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:42.053328 30475 raft_consensus.cc:729] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871b6e062d3148de9ce1dcecac8b55f8, State: Initialized, Role: FOLLOWER
I20260806 09:15:42.054095 30475 consensus_queue.cc:260] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.055294 30235 heartbeater.cc:503] Master 127.3.69.62:35703 was elected leader, sending a full tablet report...
I20260806 09:15:42.056185 30475 ts_tablet_manager.cc:1428] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260806 09:15:42.058514 30476 raft_consensus.cc:348] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.058964 30476 raft_consensus.cc:374] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:42.059121 30476 raft_consensus.cc:729] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea7b189a4d24fb985f11924cec03611, State: Initialized, Role: FOLLOWER
I20260806 09:15:42.059387 30474 raft_consensus.cc:348] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.059867 30474 raft_consensus.cc:374] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:42.059788 30476 consensus_queue.cc:260] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.059976 30474 raft_consensus.cc:729] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f03cb875e8c47ae8bcbfa3f4795b726, State: Initialized, Role: FOLLOWER
I20260806 09:15:42.060575 30474 consensus_queue.cc:260] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.061121 30445 heartbeater.cc:503] Master 127.3.69.62:35703 was elected leader, sending a full tablet report...
I20260806 09:15:42.061704 30476 ts_tablet_manager.cc:1428] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
I20260806 09:15:42.062252 30474 ts_tablet_manager.cc:1428] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260806 09:15:42.066429 30018 tablet.cc:2369] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:15:42.115046 30236 tablet.cc:2369] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:42.155965 30487 raft_consensus.cc:482] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:15:42.156303 30487 raft_consensus.cc:504] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.158499 30487 leader_election.cc:288] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 871b6e062d3148de9ce1dcecac8b55f8 (127.3.69.2:44007), eea7b189a4d24fb985f11924cec03611 (127.3.69.3:38675)
I20260806 09:15:42.165685 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:42.158702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:44007 (local address 127.3.69.1:59429)
0806 09:15:42.159574 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:42.159600 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:42.159842 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:42.160278 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:42.160283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:42.160309 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:42.160670 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.160682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.161755 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.161763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.163541 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.163554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.164896 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.164902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.165073 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.165108 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:15:42.165202 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 09:15:42.165331 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":793,"threads_started":1}
I20260806 09:15:42.165678 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:42.159024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:59429 (local address 127.3.69.2:44007)
0806 09:15:42.159287 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:42.159294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:42.159703 (+ 409us) server_negotiation.cc:407] Connection header received
0806 09:15:42.160037 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:42.160041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:42.160140 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:42.160247 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:42.160829 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.161611 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.163713 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.164668 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.164826 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.164924 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.165256 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.165395 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 09:15:42.166980 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:42.159286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:38381 (local address 127.3.69.3:38675)
0806 09:15:42.159523 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:42.159531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:42.160077 (+ 546us) server_negotiation.cc:407] Connection header received
0806 09:15:42.160295 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:42.160300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:42.160360 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:42.160458 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:42.161110 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.161826 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.164352 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.165486 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.165608 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.165713 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.166588 (+ 875us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.166733 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:15:42.166949 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250bb071458543c28df51f71ade5e056" candidate_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "871b6e062d3148de9ce1dcecac8b55f8" is_pre_election: true
I20260806 09:15:42.167012 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:42.159040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:38675 (local address 127.3.69.1:38381)
0806 09:15:42.159961 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:42.159980 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:42.160142 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:42.160482 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:42.160486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:42.160512 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:42.160950 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.160960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.161974 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.161980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.164175 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:42.164191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.166234 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.166240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:42.166424 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.166450 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 09:15:42.166561 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 09:15:42.166686 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":507,"threads_started":1}
I20260806 09:15:42.167604 30157 raft_consensus.cc:2454] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f03cb875e8c47ae8bcbfa3f4795b726 in term 0.
I20260806 09:15:42.168370 29843 leader_election.cc:302] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 871b6e062d3148de9ce1dcecac8b55f8, 8f03cb875e8c47ae8bcbfa3f4795b726; no voters:
I20260806 09:15:42.168594 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250bb071458543c28df51f71ade5e056" candidate_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea7b189a4d24fb985f11924cec03611" is_pre_election: true
I20260806 09:15:42.168850 30487 raft_consensus.cc:2781] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:15:42.168994 30487 raft_consensus.cc:482] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:15:42.169086 30487 raft_consensus.cc:3037] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:42.169188 30373 raft_consensus.cc:2454] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f03cb875e8c47ae8bcbfa3f4795b726 in term 0.
I20260806 09:15:42.172356 30487 raft_consensus.cc:504] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.173026 30487 leader_election.cc:288] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [CANDIDATE]: Term 1 election: Requested vote from peers 871b6e062d3148de9ce1dcecac8b55f8 (127.3.69.2:44007), eea7b189a4d24fb985f11924cec03611 (127.3.69.3:38675)
I20260806 09:15:42.173558 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250bb071458543c28df51f71ade5e056" candidate_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "871b6e062d3148de9ce1dcecac8b55f8"
I20260806 09:15:42.173578 30373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250bb071458543c28df51f71ade5e056" candidate_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eea7b189a4d24fb985f11924cec03611"
I20260806 09:15:42.173780 30157 raft_consensus.cc:3037] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:42.173784 30373 raft_consensus.cc:3037] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 0 FOLLOWER]: Advancing to term 1
W20260806 09:15:42.174587 30448 tablet.cc:2369] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:42.176541 30157 raft_consensus.cc:2454] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f03cb875e8c47ae8bcbfa3f4795b726 in term 1.
I20260806 09:15:42.176993 29843 leader_election.cc:302] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 871b6e062d3148de9ce1dcecac8b55f8, 8f03cb875e8c47ae8bcbfa3f4795b726; no voters:
I20260806 09:15:42.177103 30373 raft_consensus.cc:2454] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f03cb875e8c47ae8bcbfa3f4795b726 in term 1.
I20260806 09:15:42.177356 30487 raft_consensus.cc:2781] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:42.177989 30487 raft_consensus.cc:686] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 1 LEADER]: Becoming Leader. State: Replica: 8f03cb875e8c47ae8bcbfa3f4795b726, State: Running, Role: LEADER
I20260806 09:15:42.178484 30487 consensus_queue.cc:237] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } }
I20260806 09:15:42.182991 29720 catalog_manager.cc:5495] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f03cb875e8c47ae8bcbfa3f4795b726 (127.3.69.1). New cstate: current_term: 1 leader_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f03cb875e8c47ae8bcbfa3f4795b726" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 } health_report { overall_health: UNKNOWN } } }
I20260806 09:15:42.207875 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:42.201855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.3.69.1:37365)
0806 09:15:42.202036 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:42.202040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:42.202185 (+ 145us) server_negotiation.cc:407] Connection header received
0806 09:15:42.202358 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:42.202362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:42.202442 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:42.202570 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:15:42.202955 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.203781 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.205580 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.205969 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.206078 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.206690 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:15:42.207294 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:15:42.207430 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.207605 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.207647 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:15:42.216259 30157 raft_consensus.cc:1261] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 1 FOLLOWER]: Refusing update from remote peer 8f03cb875e8c47ae8bcbfa3f4795b726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:42.216310 30373 raft_consensus.cc:1261] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 1 FOLLOWER]: Refusing update from remote peer 8f03cb875e8c47ae8bcbfa3f4795b726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:42.217298 30490 consensus_queue.cc:1035] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "871b6e062d3148de9ce1dcecac8b55f8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:42.217669 30487 consensus_queue.cc:1035] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eea7b189a4d24fb985f11924cec03611" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 38675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:42.230237 30494 mvcc.cc:205] Tried to move back new op lower bound from 7315487712107663360 to 7315487711967088640. Current Snapshot: MvccSnapshot[applied={T|T < 7315487712107663360}]
I20260806 09:15:42.243330 29720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:42.243659 29720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:42.247381 29720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 226450469a8d4c1090f29978d4c03ede: Sending DeleteTablet for 3 replicas of tablet 250bb071458543c28df51f71ade5e056
I20260806 09:15:42.248588 29915 tablet_service.cc:1514] Processing DeleteTablet for tablet 250bb071458543c28df51f71ade5e056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:42 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:52708
I20260806 09:15:42.248672 30128 tablet_service.cc:1514] Processing DeleteTablet for tablet 250bb071458543c28df51f71ade5e056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:42 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37742
I20260806 09:15:42.249060 30339 tablet_service.cc:1514] Processing DeleteTablet for tablet 250bb071458543c28df51f71ade5e056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:42 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:47206
I20260806 09:15:42.255968 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 29814
I20260806 09:15:42.258836 30513 tablet_replica.cc:330] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: stopping tablet replica
I20260806 09:15:42.259191 30510 tablet_replica.cc:330] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: stopping tablet replica
I20260806 09:15:42.260079 30510 raft_consensus.cc:2229] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:15:42.260425 30513 raft_consensus.cc:2229] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:42.261005 30513 raft_consensus.cc:2258] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:42.261070 30510 raft_consensus.cc:2258] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:42.266206 30512 tablet_replica.cc:330] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: stopping tablet replica
I20260806 09:15:42.266628 30512 raft_consensus.cc:2229] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:42.266723 30510 ts_tablet_manager.cc:1905] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:42.267205 30512 raft_consensus.cc:2258] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:42.269369 30512 ts_tablet_manager.cc:1905] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:42.272444 30513 ts_tablet_manager.cc:1905] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:42.275753 30510 ts_tablet_manager.cc:1918] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:42.276003 30510 log.cc:1196] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/250bb071458543c28df51f71ade5e056
I20260806 09:15:42.276644 30510 ts_tablet_manager.cc:1939] T 250bb071458543c28df51f71ade5e056 P 8f03cb875e8c47ae8bcbfa3f4795b726: Deleting consensus metadata
I20260806 09:15:42.276973 30512 ts_tablet_manager.cc:1918] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:42.277149 30512 log.cc:1196] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/250bb071458543c28df51f71ade5e056
I20260806 09:15:42.277721 30512 ts_tablet_manager.cc:1939] T 250bb071458543c28df51f71ade5e056 P 871b6e062d3148de9ce1dcecac8b55f8: Deleting consensus metadata
I20260806 09:15:42.279050 29698 catalog_manager.cc:4841] TS 8f03cb875e8c47ae8bcbfa3f4795b726 (127.3.69.1:37365): tablet 250bb071458543c28df51f71ade5e056 (table test-table [id=64b0d2165d094a63b7134854d78ffe4c]) successfully deleted
I20260806 09:15:42.279449 29696 catalog_manager.cc:4841] TS 871b6e062d3148de9ce1dcecac8b55f8 (127.3.69.2:44007): tablet 250bb071458543c28df51f71ade5e056 (table test-table [id=64b0d2165d094a63b7134854d78ffe4c]) successfully deleted
I20260806 09:15:42.279731 30513 ts_tablet_manager.cc:1918] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:42.279912 30513 log.cc:1196] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/250bb071458543c28df51f71ade5e056
I20260806 09:15:42.280472 30513 ts_tablet_manager.cc:1939] T 250bb071458543c28df51f71ade5e056 P eea7b189a4d24fb985f11924cec03611: Deleting consensus metadata
I20260806 09:15:42.282156 29698 catalog_manager.cc:4841] TS eea7b189a4d24fb985f11924cec03611 (127.3.69.3:38675): tablet 250bb071458543c28df51f71ade5e056 (table test-table [id=64b0d2165d094a63b7134854d78ffe4c]) successfully deleted
Aug 06 09:15:42 dist-test-slave-w4v5 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:15:42.295028 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:42.259229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.3.69.1:37365)
0806 09:15:42.259472 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:42.259482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:42.269058 (+ 9576us) server_negotiation.cc:407] Connection header received
0806 09:15:42.269303 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:42.269308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:42.269397 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:42.269516 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:42.280237 (+ 10721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.281138 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.282628 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.283068 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.283273 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.287955 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:42.287973 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:42.287994 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:42.288040 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:42.292058 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:42.292750 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:42.292760 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:42.292764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:42.292983 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:42.293371 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:42.293376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:42.293379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:42.293843 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:42.294163 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.294362 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.294625 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:15:42.295831 30011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:42.398479 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 29814
I20260806 09:15:42.425572 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 30023
Aug 06 09:15:42 dist-test-slave-w4v5 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:15:42.441458 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:42.426463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.3.69.2:44007)
0806 09:15:42.426701 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:42.426706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:42.426724 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:15:42.426902 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:42.426908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:42.427000 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:42.427135 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:42.428461 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.429222 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.430587 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.431009 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.431461 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.435943 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:42.435958 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:42.435975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:42.436015 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:42.438747 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:42.439487 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:42.439495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:42.439497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:42.439670 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:42.440015 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:42.440018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:42.440020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:42.440410 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:42.440675 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.440895 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.441063 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 09:15:42.442133 30225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:42.533110 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 30023
I20260806 09:15:42.559782 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 30242
Aug 06 09:15:42 dist-test-slave-w4v5 krb5kdc[29661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:15:42.579481 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:42.560561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.3.69.3:38675)
0806 09:15:42.560796 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:42.560801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:42.560821 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:15:42.560937 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:42.560941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:42.561011 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:42.561122 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:42.562623 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.563463 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.564869 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:42.565293 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:42.565392 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:42.573989 (+ 8597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:42.574005 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:42.574022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:42.574059 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:42.576763 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:42.577483 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:42.577491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:42.577493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:42.577628 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:42.577970 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:42.577973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:42.577975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:42.578333 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:42.578559 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:42.578823 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 09:15:42.578995 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:15:42.580195 30436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:42.668562 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 30242
I20260806 09:15:42.692867 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 29676
I20260806 09:15:42.694144 29768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:42.810024 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 29676
2026-08-06T09:15:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4734 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T09:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:15:42Z Disabled control of system clock
I20260806 09:15:42.948495 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45511
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:15:43.603874 30526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:43.604331 30526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:43.604468 30526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:43.617093 30526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:43.617287 30526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:43.617343 30526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:43.617434 30526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:43.617512 30526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:43.632552 30526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:45511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:43.634538 30526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:43.637708 30526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:43.659307 30532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:43.659252 30533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:43.659124 30536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:43.661370 30526 server_base.cc:1034] running on GCE node
I20260806 09:15:43.662853 30526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:43.665331 30526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:43.666671 30526 hybrid_clock.cc:648] HybridClock initialized: now 1786007743666555 us; error 64 us; skew 500 ppm
I20260806 09:15:43.667354 30526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:43.671562 30526 webserver.cc:460] Webserver started at http://127.3.69.62:39291/ using document root <none> and password file <none>
I20260806 09:15:43.672842 30526 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:43.673064 30526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:43.673635 30526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:43.679877 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "ae8cd62a4557411282c88ed160d85fda"
format_stamp: "Formatted at 2026-08-06 09:15:43 on dist-test-slave-w4v5"
server_key: "6918d67a8e2b779a972f0082f49eee49"
server_key_iv: "ffe6de4a3d4e34b07a4558e1d26ea33d"
server_key_version: "encryptionkey@0"
I20260806 09:15:43.681500 30526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "ae8cd62a4557411282c88ed160d85fda"
format_stamp: "Formatted at 2026-08-06 09:15:43 on dist-test-slave-w4v5"
server_key: "6918d67a8e2b779a972f0082f49eee49"
server_key_iv: "ffe6de4a3d4e34b07a4558e1d26ea33d"
server_key_version: "encryptionkey@0"
I20260806 09:15:43.701385 30526 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.013s sys 0.004s
I20260806 09:15:43.712384 30545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:43.714895 30526 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.008s sys 0.000s
I20260806 09:15:43.715250 30526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "ae8cd62a4557411282c88ed160d85fda"
format_stamp: "Formatted at 2026-08-06 09:15:43 on dist-test-slave-w4v5"
server_key: "6918d67a8e2b779a972f0082f49eee49"
server_key_iv: "ffe6de4a3d4e34b07a4558e1d26ea33d"
server_key_version: "encryptionkey@0"
I20260806 09:15:43.715670 30526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:43.759819 30526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:43.761445 30526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:43.761972 30526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:43.799528 30526 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:45511
I20260806 09:15:43.799542 30629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:45511 every 8 connection(s)
I20260806 09:15:43.802228 30526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:43.809643 30630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:43.810946 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 30526
I20260806 09:15:43.811348 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:15:43.812671 3348 external_mini_cluster.cc:1428] Setting key 4332fc50a4015db0bd052aa8deb4c463
I20260806 09:15:43.835163 30630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda: Bootstrap starting.
I20260806 09:15:43.840759 30633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:43.817162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.3.69.62:45511)
0806 09:15:43.819399 (+ 2237us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:43.819428 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:43.819510 (+ 82us) server_negotiation.cc:407] Connection header received
0806 09:15:43.823489 (+ 3979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:43.823535 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:43.824729 (+ 1194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:43.825651 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:43.826858 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:43.829169 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:43.831876 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:43.833162 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:43.833570 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:43.835995 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:43.836012 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:43.836036 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:43.837011 (+ 975us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:43.837171 (+ 160us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:43.837214 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:43.837490 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:43.838013 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 09:15:43.838610 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":801,"threads_started":1}
I20260806 09:15:43.841718 30630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:43.844121 30630 log.cc:824] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:43.851894 30630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda: No bootstrap required, opened a new log
I20260806 09:15:43.861078 30630 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8cd62a4557411282c88ed160d85fda" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45511 } }
I20260806 09:15:43.861791 30630 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:43.861909 30630 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8cd62a4557411282c88ed160d85fda, State: Initialized, Role: FOLLOWER
I20260806 09:15:43.862545 30630 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8cd62a4557411282c88ed160d85fda" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45511 } }
I20260806 09:15:43.862824 30630 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:43.862967 30630 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:43.863243 30630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:43.865581 30630 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8cd62a4557411282c88ed160d85fda" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45511 } }
I20260806 09:15:43.866145 30630 leader_election.cc:302] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8cd62a4557411282c88ed160d85fda; no voters:
I20260806 09:15:43.866768 30630 leader_election.cc:288] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:15:43.866933 30637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:43.867833 30637 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [term 1 LEADER]: Becoming Leader. State: Replica: ae8cd62a4557411282c88ed160d85fda, State: Running, Role: LEADER
I20260806 09:15:43.868482 30637 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8cd62a4557411282c88ed160d85fda" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45511 } }
I20260806 09:15:43.869213 30630 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:43.871309 30639 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8cd62a4557411282c88ed160d85fda. Latest consensus state: current_term: 1 leader_uuid: "ae8cd62a4557411282c88ed160d85fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8cd62a4557411282c88ed160d85fda" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45511 } } }
I20260806 09:15:43.871848 30639 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:43.871896 30638 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae8cd62a4557411282c88ed160d85fda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8cd62a4557411282c88ed160d85fda" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 45511 } } }
I20260806 09:15:43.872831 30638 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:43.876375 30642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:43.881824 30642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:43.891405 30642 catalog_manager.cc:1347] Generated new cluster ID: 476173bfded54ffdb15964412f2c1987
I20260806 09:15:43.891556 30642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:43.926586 30642 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:15:43.929163 30642 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:43.948724 30642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda: Generated new TSK 0
I20260806 09:15:43.951814 30642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:15:43.972216 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45511
--builtin_ntp_servers=127.3.69.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:15:44.512255 30666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:44.512586 30666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:44.512768 30666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:44.519788 30666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:44.519919 30666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:44.520097 30666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:15:44.529778 30666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:44.531100 30666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:44.532893 30666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:44.546474 30673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:44.546427 30677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:44.546371 30674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:44.547245 30666 server_base.cc:1034] running on GCE node
I20260806 09:15:44.547917 30666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:44.548902 30666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:44.550143 30666 hybrid_clock.cc:648] HybridClock initialized: now 1786007744550090 us; error 62 us; skew 500 ppm
I20260806 09:15:44.550490 30666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:44.552780 30666 webserver.cc:460] Webserver started at http://127.3.69.1:40551/ using document root <none> and password file <none>
I20260806 09:15:44.553467 30666 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:44.553604 30666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:44.553946 30666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:44.556809 30666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "b469df01316f419cb123f319cba826a6"
format_stamp: "Formatted at 2026-08-06 09:15:44 on dist-test-slave-w4v5"
server_key: "31a0dd8f38e14772db80cda7264b9f24"
server_key_iv: "7166425cc0f00b588724d5c1ede90f07"
server_key_version: "encryptionkey@0"
I20260806 09:15:44.557648 30666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "b469df01316f419cb123f319cba826a6"
format_stamp: "Formatted at 2026-08-06 09:15:44 on dist-test-slave-w4v5"
server_key: "31a0dd8f38e14772db80cda7264b9f24"
server_key_iv: "7166425cc0f00b588724d5c1ede90f07"
server_key_version: "encryptionkey@0"
I20260806 09:15:44.562799 30666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 09:15:44.567320 30685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:44.568632 30666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:15:44.568822 30666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "b469df01316f419cb123f319cba826a6"
format_stamp: "Formatted at 2026-08-06 09:15:44 on dist-test-slave-w4v5"
server_key: "31a0dd8f38e14772db80cda7264b9f24"
server_key_iv: "7166425cc0f00b588724d5c1ede90f07"
server_key_version: "encryptionkey@0"
I20260806 09:15:44.569048 30666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:44.599730 30666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:44.600626 30666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:44.600934 30666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:44.601956 30666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:44.603543 30666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:44.603674 30666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:44.603812 30666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:44.603904 30666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:44.638190 30666 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:43101
I20260806 09:15:44.638278 30848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:43101 every 8 connection(s)
I20260806 09:15:44.639550 30666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:15:44.649019 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 30666
I20260806 09:15:44.649210 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:15:44.649772 3348 external_mini_cluster.cc:1428] Setting key 1b8af7a512cb6d58f1aae78d0c61b50e
I20260806 09:15:44.652988 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45511
--builtin_ntp_servers=127.3.69.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:15:44.653154 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:44.642201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:53147 (local address 127.3.69.62:45511)
0806 09:15:44.642678 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:44.642683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:44.643895 (+ 1212us) server_negotiation.cc:407] Connection header received
0806 09:15:44.645710 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:44.645716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:44.645807 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:44.645936 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:44.646978 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:44.647845 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:44.649982 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:44.650504 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:44.650643 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:44.652262 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:44.652267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:44.652270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:44.652287 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:44.652328 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:44.652341 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:44.652421 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:44.652766 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 09:15:44.652871 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":247,"threads_started":1}
I20260806 09:15:44.654546 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:44.642793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45511 (local address 127.3.69.1:53147)
0806 09:15:44.643735 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:44.643770 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:44.645425 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:44.646167 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:44.646175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:44.646256 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:44.646803 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:44.646825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:44.647997 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:44.648007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:44.649750 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:44.649769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:44.650753 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:44.650761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:44.651036 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:44.651868 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:44.651872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:44.651937 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:44.652032 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:44.652454 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:44.652467 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:44.652477 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:15:44.652718 (+ 241us) client_negotiation.cc:240] Negotiation successful
0806 09:15:44.653103 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":337,"threads_started":1}
I20260806 09:15:44.656167 30849 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45511
I20260806 09:15:44.656495 30849 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:44.657392 30849 heartbeater.cc:511] Master 127.3.69.62:45511 requested a full tablet report, sending...
I20260806 09:15:44.659569 30569 ts_manager.cc:194] Registered new tserver with Master: b469df01316f419cb123f319cba826a6 (127.3.69.1:43101)
I20260806 09:15:44.661062 30569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.1:53147
W20260806 09:15:45.008924 30857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:45.009204 30857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:45.009347 30857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:45.016718 30857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:45.016824 30857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:45.016995 30857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:15:45.026350 30857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:45.027719 30857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:45.029520 30857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:45.043438 30866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:45.043638 30869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:45.043931 30867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:45.044338 30857 server_base.cc:1034] running on GCE node
I20260806 09:15:45.044972 30857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:45.045941 30857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:45.047188 30857 hybrid_clock.cc:648] HybridClock initialized: now 1786007745047133 us; error 59 us; skew 500 ppm
I20260806 09:15:45.047523 30857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:45.049777 30857 webserver.cc:460] Webserver started at http://127.3.69.2:42521/ using document root <none> and password file <none>
I20260806 09:15:45.050640 30857 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:45.050776 30857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:45.051218 30857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:45.054150 30857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8"
format_stamp: "Formatted at 2026-08-06 09:15:45 on dist-test-slave-w4v5"
server_key: "5243b35cca8353e0534c8264c2245bd4"
server_key_iv: "e0bf7b9586e643d5c55e7243a67f2409"
server_key_version: "encryptionkey@0"
I20260806 09:15:45.054960 30857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8"
format_stamp: "Formatted at 2026-08-06 09:15:45 on dist-test-slave-w4v5"
server_key: "5243b35cca8353e0534c8264c2245bd4"
server_key_iv: "e0bf7b9586e643d5c55e7243a67f2409"
server_key_version: "encryptionkey@0"
I20260806 09:15:45.060065 30857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:15:45.063442 30879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:45.064692 30857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:15:45.064880 30857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8"
format_stamp: "Formatted at 2026-08-06 09:15:45 on dist-test-slave-w4v5"
server_key: "5243b35cca8353e0534c8264c2245bd4"
server_key_iv: "e0bf7b9586e643d5c55e7243a67f2409"
server_key_version: "encryptionkey@0"
I20260806 09:15:45.065110 30857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:45.084933 30857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:45.085745 30857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:45.086042 30857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:45.087034 30857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:45.088672 30857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:45.088761 30857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:45.088835 30857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:45.088891 30857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:45.123651 30857 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:41015
I20260806 09:15:45.123736 31051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:41015 every 8 connection(s)
I20260806 09:15:45.124981 30857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:15:45.128299 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 30857
I20260806 09:15:45.128517 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:15:45.129168 3348 external_mini_cluster.cc:1428] Setting key 78699976e0a979ca7966a84ee80e71fe
I20260806 09:15:45.132547 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45511
--builtin_ntp_servers=127.3.69.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:15:45.139757 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.127766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:47001 (local address 127.3.69.62:45511)
0806 09:15:45.128017 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.128025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.129657 (+ 1632us) server_negotiation.cc:407] Connection header received
0806 09:15:45.131959 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.131964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.132029 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.132116 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:45.133600 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.134306 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.136799 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.137193 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.137291 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.138960 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:45.138965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:45.138969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:45.138986 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:45.139018 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:45.139031 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:45.139186 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.139424 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.139502 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:15:45.141232 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.128274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45511 (local address 127.3.69.2:47001)
0806 09:15:45.129459 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:45.129507 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:45.131658 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:45.132389 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:45.132402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:45.132516 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:45.133408 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.133436 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.134478 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.134486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.136649 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.136668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.137435 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.137446 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.137786 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.138587 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:45.138593 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:45.138655 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:45.138761 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:45.139191 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:45.139206 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:45.139218 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:15:45.139410 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 09:15:45.139838 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":384,"threads_started":1}
I20260806 09:15:45.142692 31053 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45511
I20260806 09:15:45.143036 31053 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:45.143990 31053 heartbeater.cc:511] Master 127.3.69.62:45511 requested a full tablet report, sending...
I20260806 09:15:45.145671 30569 ts_manager.cc:194] Registered new tserver with Master: 25fb23a9f8c64e9c8c10c05cf3d5a8a8 (127.3.69.2:41015)
I20260806 09:15:45.146925 30569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.2:47001
W20260806 09:15:45.501077 31061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:45.501425 31061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:45.501590 31061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:45.508735 31061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:45.508875 31061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:45.509050 31061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:15:45.518066 31061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:45.519399 31061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:45.521312 31061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:45.535410 31069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:45.535354 31071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:45.535395 31068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:45.536916 31061 server_base.cc:1034] running on GCE node
I20260806 09:15:45.537653 31061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:45.538614 31061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:45.539858 31061 hybrid_clock.cc:648] HybridClock initialized: now 1786007745539812 us; error 56 us; skew 500 ppm
I20260806 09:15:45.540198 31061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:45.542508 31061 webserver.cc:460] Webserver started at http://127.3.69.3:39443/ using document root <none> and password file <none>
I20260806 09:15:45.543282 31061 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:45.543416 31061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:45.543748 31061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:45.546587 31061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "8dfebd05439d4405a3162fed238d52ec"
format_stamp: "Formatted at 2026-08-06 09:15:45 on dist-test-slave-w4v5"
server_key: "78825dfd722cb100e06822714022d14c"
server_key_iv: "b1057cd08e352010de234904430ae3e6"
server_key_version: "encryptionkey@0"
I20260806 09:15:45.547463 31061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "8dfebd05439d4405a3162fed238d52ec"
format_stamp: "Formatted at 2026-08-06 09:15:45 on dist-test-slave-w4v5"
server_key: "78825dfd722cb100e06822714022d14c"
server_key_iv: "b1057cd08e352010de234904430ae3e6"
server_key_version: "encryptionkey@0"
I20260806 09:15:45.552412 31061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 09:15:45.556020 31079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:45.557368 31061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:15:45.557549 31061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "8dfebd05439d4405a3162fed238d52ec"
format_stamp: "Formatted at 2026-08-06 09:15:45 on dist-test-slave-w4v5"
server_key: "78825dfd722cb100e06822714022d14c"
server_key_iv: "b1057cd08e352010de234904430ae3e6"
server_key_version: "encryptionkey@0"
I20260806 09:15:45.557777 31061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:45.575805 31061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:45.576714 31061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:45.576995 31061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:45.578037 31061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:45.579753 31061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:45.579911 31061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:45.580040 31061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:45.580132 31061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:45.617842 31061 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:41501
I20260806 09:15:45.617902 31246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:41501 every 8 connection(s)
I20260806 09:15:45.619256 31061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:15:45.627844 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 31061
I20260806 09:15:45.628051 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:15:45.628585 3348 external_mini_cluster.cc:1428] Setting key 52a877d758069b2aca42085b6a08fb66
I20260806 09:15:45.632563 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.621875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:57449 (local address 127.3.69.62:45511)
0806 09:15:45.622106 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.622113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.623588 (+ 1475us) server_negotiation.cc:407] Connection header received
0806 09:15:45.625552 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.625557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.625648 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.625746 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:45.626816 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.627853 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.629708 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.630243 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.630370 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.631869 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:45.631874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:45.631877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:45.631894 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:45.631925 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:45.631937 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:45.632010 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.632212 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.632288 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:15:45.633526 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.622356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:45511 (local address 127.3.69.3:57449)
0806 09:15:45.623414 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:45.623451 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:45.625273 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:45.625945 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:45.625952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:45.626034 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:45.626613 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.626630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.628061 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.628065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.629530 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.629542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.630496 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.630505 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.630736 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.631572 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:45.631575 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:45.631617 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:45.631691 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:45.632030 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:45.632038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:45.632044 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:15:45.632190 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 09:15:45.632459 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":337,"threads_started":1}
I20260806 09:15:45.634804 31247 heartbeater.cc:347] Connected to a master server at 127.3.69.62:45511
I20260806 09:15:45.635172 31247 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:45.635931 31247 heartbeater.cc:511] Master 127.3.69.62:45511 requested a full tablet report, sending...
I20260806 09:15:45.637251 30569 ts_manager.cc:194] Registered new tserver with Master: 8dfebd05439d4405a3162fed238d52ec (127.3.69.3:41501)
I20260806 09:15:45.638445 30569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.3:57449
I20260806 09:15:45.643839 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:15:45.654531 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.648609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.3.69.62:45511)
0806 09:15:45.648876 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.648881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.649243 (+ 362us) server_negotiation.cc:407] Connection header received
0806 09:15:45.649367 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.649371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.649430 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.649510 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:45.649998 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.650727 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.652161 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.652561 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.652653 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.653689 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:45.653707 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:45.653710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:45.653726 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:45.653755 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:45.653768 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:45.653878 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.654040 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.654116 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 09:15:45.664533 30849 heartbeater.cc:503] Master 127.3.69.62:45511 was elected leader, sending a full tablet report...
I20260806 09:15:45.658730 30569 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:15:45.666607 30569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:15:45.692816 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.682893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.3.69.1:43101)
0806 09:15:45.684082 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.684095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.684133 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:15:45.684251 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.684263 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.685024 (+ 761us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.685300 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:45.686034 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.687574 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.690572 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.691625 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.691788 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.692078 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.692279 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.692449 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":593,"threads_started":1}
I20260806 09:15:45.692843 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.681576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58302)
0806 09:15:45.682284 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:45.682307 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:45.682453 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:45.685316 (+ 2863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:45.685323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:45.685353 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:45.685833 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.685851 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.687748 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.687754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.690274 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.690291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.691818 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.691825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.692007 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.692067 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 09:15:45.692235 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 09:15:45.692422 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":1,"thread_start_us":413,"threads_started":1}
I20260806 09:15:45.695020 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.682364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41501 (local address 127.0.0.1:58698)
0806 09:15:45.682990 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:45.683009 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:45.683157 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:45.685761 (+ 2604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:45.685769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:45.685811 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:45.686372 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.686387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.688322 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.688328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.690464 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.690491 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.694111 (+ 3620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.694122 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.694298 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.694333 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:15:45.694435 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 09:15:45.694599 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":984,"threads_started":1}
I20260806 09:15:45.695315 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.683608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.3.69.3:41501)
0806 09:15:45.685136 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.685148 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.685174 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:15:45.685281 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.685288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.685503 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.685722 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:45.686592 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.688124 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.690707 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.693764 (+ 3057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.693923 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.694265 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.694558 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.694804 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":1231,"threads_started":1}
I20260806 09:15:45.697461 30750 tablet_service.cc:1467] Processing CreateTablet for tablet 25268ad3e01c4f2abef9c7355ba79854 (DEFAULT_TABLE table=test-table [id=d9d84382f9064b34badc7ececf33b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:45.699170 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25268ad3e01c4f2abef9c7355ba79854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:45.699342 31151 tablet_service.cc:1467] Processing CreateTablet for tablet 25268ad3e01c4f2abef9c7355ba79854 (DEFAULT_TABLE table=test-table [id=d9d84382f9064b34badc7ececf33b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:45.700856 31151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25268ad3e01c4f2abef9c7355ba79854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:45.702556 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.680914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:41015 (local address 127.0.0.1:42354)
0806 09:15:45.681956 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:45.682009 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:45.682200 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:45.694844 (+ 12644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:45.694854 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:45.694882 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:45.695369 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.695385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.697534 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.697540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.699733 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.699749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.701688 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.701698 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.701887 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.701926 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:15:45.702026 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 09:15:45.702168 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":594,"threads_started":1}
I20260806 09:15:45.702669 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.680924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.3.69.2:41015)
0806 09:15:45.694145 (+ 13221us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.694157 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.694189 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:15:45.694339 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.694346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.694593 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.695606 (+ 1013us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:45.695866 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.697346 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.699989 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.701474 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.701621 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.701868 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.702093 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.702296 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13054,"thread_start_us":13356,"threads_started":1}
I20260806 09:15:45.706638 30956 tablet_service.cc:1467] Processing CreateTablet for tablet 25268ad3e01c4f2abef9c7355ba79854 (DEFAULT_TABLE table=test-table [id=d9d84382f9064b34badc7ececf33b943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:45.708098 30956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25268ad3e01c4f2abef9c7355ba79854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:45.713207 31272 tablet_bootstrap.cc:492] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: Bootstrap starting.
I20260806 09:15:45.716012 31273 tablet_bootstrap.cc:492] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Bootstrap starting.
I20260806 09:15:45.717127 31272 tablet_bootstrap.cc:654] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:45.718745 31272 log.cc:824] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:45.719880 31273 tablet_bootstrap.cc:654] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:45.721279 31273 log.cc:824] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:45.722098 31272 tablet_bootstrap.cc:492] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: No bootstrap required, opened a new log
I20260806 09:15:45.722404 31277 tablet_bootstrap.cc:492] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Bootstrap starting.
I20260806 09:15:45.722453 31272 ts_tablet_manager.cc:1397] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260806 09:15:45.727356 31277 tablet_bootstrap.cc:654] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:45.728744 31277 log.cc:824] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:45.730078 31273 tablet_bootstrap.cc:492] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: No bootstrap required, opened a new log
I20260806 09:15:45.730420 31273 ts_tablet_manager.cc:1397] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260806 09:15:45.732057 31277 tablet_bootstrap.cc:492] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: No bootstrap required, opened a new log
I20260806 09:15:45.731880 31272 raft_consensus.cc:348] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.732424 31277 ts_tablet_manager.cc:1397] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260806 09:15:45.732453 31272 raft_consensus.cc:374] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:45.732626 31272 raft_consensus.cc:729] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b469df01316f419cb123f319cba826a6, State: Initialized, Role: FOLLOWER
I20260806 09:15:45.733266 31272 consensus_queue.cc:260] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.735337 31272 ts_tablet_manager.cc:1428] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260806 09:15:45.742142 31273 raft_consensus.cc:348] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.742648 31273 raft_consensus.cc:374] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:45.742754 31273 raft_consensus.cc:729] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfebd05439d4405a3162fed238d52ec, State: Initialized, Role: FOLLOWER
I20260806 09:15:45.743454 31273 consensus_queue.cc:260] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.743868 31277 raft_consensus.cc:348] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.744369 31277 raft_consensus.cc:374] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:45.744522 31277 raft_consensus.cc:729] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fb23a9f8c64e9c8c10c05cf3d5a8a8, State: Initialized, Role: FOLLOWER
I20260806 09:15:45.744657 31247 heartbeater.cc:503] Master 127.3.69.62:45511 was elected leader, sending a full tablet report...
I20260806 09:15:45.745234 31277 consensus_queue.cc:260] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.745569 31273 ts_tablet_manager.cc:1428] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260806 09:15:45.746527 31053 heartbeater.cc:503] Master 127.3.69.62:45511 was elected leader, sending a full tablet report...
I20260806 09:15:45.747289 31277 ts_tablet_manager.cc:1428] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260806 09:15:45.817960 31283 raft_consensus.cc:482] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:15:45.818365 31283 raft_consensus.cc:504] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.820317 31283 leader_election.cc:288] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25fb23a9f8c64e9c8c10c05cf3d5a8a8 (127.3.69.2:41015), 8dfebd05439d4405a3162fed238d52ec (127.3.69.3:41501)
I20260806 09:15:45.827963 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.820776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:35041 (local address 127.3.69.2:41015)
0806 09:15:45.820967 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.820972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.821458 (+ 486us) server_negotiation.cc:407] Connection header received
0806 09:15:45.821688 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.821717 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.821834 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.821979 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:45.822715 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.823726 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.826193 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.826956 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.827130 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.827255 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.827553 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.827678 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:15:45.828049 31288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.820623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:41015 (local address 127.3.69.1:35041)
0806 09:15:45.821331 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:45.821353 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:45.821515 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:45.822018 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:45.822022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:45.822054 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:45.822532 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.822544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.823858 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.823866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.825939 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.825951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.827201 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.827208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.827383 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.827407 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:15:45.827524 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 09:15:45.827663 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":84,"thread_start_us":385,"threads_started":1}
I20260806 09:15:45.828248 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.821647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:49089 (local address 127.3.69.3:41501)
0806 09:15:45.821845 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.821851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.822325 (+ 474us) server_negotiation.cc:407] Connection header received
0806 09:15:45.822545 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.822549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.822621 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.822733 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:45.823442 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.824311 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.826305 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.827280 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.827423 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.827538 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.827865 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.827993 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:15:45.828567 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.821477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41501 (local address 127.3.69.1:49089)
0806 09:15:45.822193 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:45.822207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:45.822379 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:45.822750 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:45.822755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:45.822779 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:45.823266 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.823277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.824470 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.824480 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.826094 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:45.826109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.827507 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.827515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:45.827685 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.827708 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 09:15:45.827816 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 09:15:45.827943 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":85,"thread_start_us":387,"threads_started":1}
I20260806 09:15:45.829481 30985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25268ad3e01c4f2abef9c7355ba79854" candidate_uuid: "b469df01316f419cb123f319cba826a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" is_pre_election: true
I20260806 09:15:45.830015 30985 raft_consensus.cc:2454] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b469df01316f419cb123f319cba826a6 in term 0.
I20260806 09:15:45.829774 31176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25268ad3e01c4f2abef9c7355ba79854" candidate_uuid: "b469df01316f419cb123f319cba826a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfebd05439d4405a3162fed238d52ec" is_pre_election: true
I20260806 09:15:45.830453 31176 raft_consensus.cc:2454] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b469df01316f419cb123f319cba826a6 in term 0.
I20260806 09:15:45.830778 30688 leader_election.cc:302] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25fb23a9f8c64e9c8c10c05cf3d5a8a8, b469df01316f419cb123f319cba826a6; no voters:
I20260806 09:15:45.831255 31283 raft_consensus.cc:2781] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:15:45.831421 31283 raft_consensus.cc:482] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:15:45.831553 31283 raft_consensus.cc:3037] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:45.833846 31283 raft_consensus.cc:504] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.834473 31283 leader_election.cc:288] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [CANDIDATE]: Term 1 election: Requested vote from peers 25fb23a9f8c64e9c8c10c05cf3d5a8a8 (127.3.69.2:41015), 8dfebd05439d4405a3162fed238d52ec (127.3.69.3:41501)
I20260806 09:15:45.834968 30985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25268ad3e01c4f2abef9c7355ba79854" candidate_uuid: "b469df01316f419cb123f319cba826a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8"
I20260806 09:15:45.835170 31176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25268ad3e01c4f2abef9c7355ba79854" candidate_uuid: "b469df01316f419cb123f319cba826a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfebd05439d4405a3162fed238d52ec"
I20260806 09:15:45.835212 30985 raft_consensus.cc:3037] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:45.835382 31176 raft_consensus.cc:3037] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:45.838207 31176 raft_consensus.cc:2454] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b469df01316f419cb123f319cba826a6 in term 1.
I20260806 09:15:45.838460 30985 raft_consensus.cc:2454] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b469df01316f419cb123f319cba826a6 in term 1.
I20260806 09:15:45.839053 30688 leader_election.cc:302] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfebd05439d4405a3162fed238d52ec, b469df01316f419cb123f319cba826a6; no voters:
I20260806 09:15:45.839437 31283 raft_consensus.cc:2781] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:45.839890 31283 raft_consensus.cc:686] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 1 LEADER]: Becoming Leader. State: Replica: b469df01316f419cb123f319cba826a6, State: Running, Role: LEADER
I20260806 09:15:45.840391 31283 consensus_queue.cc:237] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } }
I20260806 09:15:45.845463 30567 catalog_manager.cc:5495] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b469df01316f419cb123f319cba826a6 (127.3.69.1). New cstate: current_term: 1 leader_uuid: "b469df01316f419cb123f319cba826a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b469df01316f419cb123f319cba826a6" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 43101 } health_report { overall_health: HEALTHY } } }
W20260806 09:15:45.871105 31248 tablet.cc:2369] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:15:45.877194 31054 tablet.cc:2369] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:45.887043 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.881621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.3.69.1:43101)
0806 09:15:45.881813 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.881818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.881935 (+ 117us) server_negotiation.cc:407] Connection header received
0806 09:15:45.882115 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.882119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.882228 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.882353 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:15:45.882708 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.883491 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.884965 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.885378 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.885469 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.886023 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:15:45.886530 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:15:45.886658 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.886795 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.886833 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 09:15:45.891906 30854 tablet.cc:2369] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:45.895505 31176 raft_consensus.cc:1261] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 1 FOLLOWER]: Refusing update from remote peer b469df01316f419cb123f319cba826a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:45.895610 30985 raft_consensus.cc:1261] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 1 FOLLOWER]: Refusing update from remote peer b469df01316f419cb123f319cba826a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:45.896617 31296 consensus_queue.cc:1035] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfebd05439d4405a3162fed238d52ec" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:45.897018 31283 consensus_queue.cc:1035] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25fb23a9f8c64e9c8c10c05cf3d5a8a8" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:45.925020 30567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:45.925311 30567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:45.928817 30567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae8cd62a4557411282c88ed160d85fda: Sending DeleteTablet for 3 replicas of tablet 25268ad3e01c4f2abef9c7355ba79854
I20260806 09:15:45.930143 30956 tablet_service.cc:1514] Processing DeleteTablet for tablet 25268ad3e01c4f2abef9c7355ba79854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:45 UTC) from {username='slave'} at 127.0.0.1:42354
I20260806 09:15:45.930431 31151 tablet_service.cc:1514] Processing DeleteTablet for tablet 25268ad3e01c4f2abef9c7355ba79854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:45 UTC) from {username='slave'} at 127.0.0.1:58698
I20260806 09:15:45.930930 30750 tablet_service.cc:1514] Processing DeleteTablet for tablet 25268ad3e01c4f2abef9c7355ba79854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:45 UTC) from {username='slave'} at 127.0.0.1:58302
I20260806 09:15:45.935292 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 30666
I20260806 09:15:45.943742 31311 tablet_replica.cc:330] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: stopping tablet replica
I20260806 09:15:45.944134 31311 raft_consensus.cc:2229] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:45.944687 31311 raft_consensus.cc:2258] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:45.947285 31311 ts_tablet_manager.cc:1905] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:45.948547 31313 tablet_replica.cc:330] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: stopping tablet replica
I20260806 09:15:45.948973 31313 raft_consensus.cc:2229] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:45.949465 31313 raft_consensus.cc:2258] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:45.951562 31314 tablet_replica.cc:330] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: stopping tablet replica
I20260806 09:15:45.951769 31313 ts_tablet_manager.cc:1905] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:45.952006 31314 raft_consensus.cc:2229] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:15:45.952713 31314 raft_consensus.cc:2258] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:45.954443 31311 ts_tablet_manager.cc:1918] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:45.954680 31311 log.cc:1196] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/25268ad3e01c4f2abef9c7355ba79854
I20260806 09:15:45.955194 31270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:45.936859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.3.69.1:43101)
0806 09:15:45.937027 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:45.937033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:45.940294 (+ 3261us) server_negotiation.cc:407] Connection header received
0806 09:15:45.941114 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:45.941120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:45.941234 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:45.941311 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:45.943050 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.944288 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.950523 (+ 6235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:45.951301 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:45.951426 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:45.953081 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:45.953093 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:45.953111 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:45.953982 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:45.954040 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:45.954065 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:45.954182 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:45.954635 (+ 453us) server_negotiation.cc:299] Negotiation successful
0806 09:15:45.954755 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 09:15:45.955402 31311 ts_tablet_manager.cc:1939] T 25268ad3e01c4f2abef9c7355ba79854 P 25fb23a9f8c64e9c8c10c05cf3d5a8a8: Deleting consensus metadata
I20260806 09:15:45.955991 30839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:45.957273 30548 catalog_manager.cc:4841] TS 25fb23a9f8c64e9c8c10c05cf3d5a8a8 (127.3.69.2:41015): tablet 25268ad3e01c4f2abef9c7355ba79854 (table test-table [id=d9d84382f9064b34badc7ececf33b943]) successfully deleted
I20260806 09:15:45.959038 31313 ts_tablet_manager.cc:1918] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:45.959255 31313 log.cc:1196] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/25268ad3e01c4f2abef9c7355ba79854
I20260806 09:15:45.959805 31313 ts_tablet_manager.cc:1939] T 25268ad3e01c4f2abef9c7355ba79854 P 8dfebd05439d4405a3162fed238d52ec: Deleting consensus metadata
I20260806 09:15:45.961609 30548 catalog_manager.cc:4841] TS 8dfebd05439d4405a3162fed238d52ec (127.3.69.3:41501): tablet 25268ad3e01c4f2abef9c7355ba79854 (table test-table [id=d9d84382f9064b34badc7ececf33b943]) successfully deleted
I20260806 09:15:45.957186 31314 ts_tablet_manager.cc:1905] T 25268ad3e01c4f2abef9c7355ba79854 P b469df01316f419cb123f319cba826a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:46.053758 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 30666
I20260806 09:15:46.082203 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 30857
W20260806 09:15:46.082134 30550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.69.1:43101 (error 108)
I20260806 09:15:46.084242 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.083769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58312)
0806 09:15:46.083957 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:46.084061 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260806 09:15:46.084501 30550 catalog_manager.cc:4584] TS b469df01316f419cb123f319cba826a6 (127.3.69.1:43101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25268ad3e01c4f2abef9c7355ba79854: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
I20260806 09:15:46.092490 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.083155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.3.69.2:41015)
0806 09:15:46.083394 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:46.083400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:46.083423 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:15:46.083504 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:46.083510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:46.083613 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:46.083700 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:46.084241 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:46.085117 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:46.086458 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:46.086841 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:46.086930 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:46.091156 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:46.091165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:46.091178 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:46.091744 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:46.091791 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:46.091811 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:46.091900 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:46.092027 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 09:15:46.092116 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:15:46.093174 31045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:46.138228 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.137664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58316)
0806 09:15:46.137914 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:46.138023 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 09:15:46.139330 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.138896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58330)
0806 09:15:46.139069 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:46.139181 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 09:15:46.199934 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.199377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58332)
0806 09:15:46.199628 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:46.199744 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:15:46.201038 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.200614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58338)
0806 09:15:46.200774 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:46.200857 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 09:15:46.203009 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 30857
I20260806 09:15:46.229423 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 31061
I20260806 09:15:46.236840 31271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.230247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.3.69.3:41501)
0806 09:15:46.230453 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:46.230460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:46.230495 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:15:46.230704 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:46.230714 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:46.230803 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:46.230915 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:46.231451 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:46.232218 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:46.233547 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:46.233969 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:46.234111 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:46.235457 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:46.235470 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:46.235491 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:46.236083 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:46.236136 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:46.236157 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:46.236251 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:46.236373 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 09:15:46.236468 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:15:46.237555 31238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:46.282213 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.281605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58350)
0806 09:15:46.281860 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:46.281974 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 09:15:46.283416 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.282929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58364)
0806 09:15:46.283180 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:46.283275 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 09:15:46.327498 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 31061
I20260806 09:15:46.353987 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 30526
I20260806 09:15:46.355261 30624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:46.468057 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:46.467325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:43101 (local address 127.0.0.1:58372)
0806 09:15:46.467664 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:46.467769 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:43101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260806 09:15:46.468197 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 30526
2026-08-06T09:15:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3654 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:15:46 dist-test-slave-w4v5 krb5kdc[31324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:15:46 dist-test-slave-w4v5 krb5kdc[31324](info): set up 2 sockets
Aug 06 09:15:46 dist-test-slave-w4v5 krb5kdc[31324](info): commencing operation
krb5kdc: starting...
W20260806 09:15:47.812178 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.002s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:15:48 dist-test-slave-w4v5 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:15:48.130077 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36321
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:36321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:48.472107 31350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:48.472388 31350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:48.472564 31350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:48.479669 31350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:48.479768 31350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:48.479873 31350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:48.479974 31350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:48.480058 31350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:48.489806 31350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:36321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:48.491153 31350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:48.492477 31350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:48.504230 31358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:48.504195 31361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:48.504163 31365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:48.505286 31350 server_base.cc:1034] running on GCE node
I20260806 09:15:48.506069 31350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:48.507457 31350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:48.508668 31350 hybrid_clock.cc:648] HybridClock initialized: now 1786007748508619 us; error 33 us; skew 500 ppm
I20260806 09:15:48.509025 31350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:48 dist-test-slave-w4v5 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:48.512100 31350 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:15:48.513525 31350 webserver.cc:460] Webserver started at http://127.3.69.62:41217/ using document root <none> and password file <none>
I20260806 09:15:48.514145 31350 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:48.514246 31350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:48.514494 31350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:48.517421 31350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "dc91a6ae9a704e9cb20b135568f8558f"
format_stamp: "Formatted at 2026-08-06 09:15:48 on dist-test-slave-w4v5"
server_key: "4342565bdf1a68c4e979a6b67256fe08"
server_key_iv: "66618d4e0d9d6ac31be8968c9f698a62"
server_key_version: "encryptionkey@0"
I20260806 09:15:48.518118 31350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "dc91a6ae9a704e9cb20b135568f8558f"
format_stamp: "Formatted at 2026-08-06 09:15:48 on dist-test-slave-w4v5"
server_key: "4342565bdf1a68c4e979a6b67256fe08"
server_key_iv: "66618d4e0d9d6ac31be8968c9f698a62"
server_key_version: "encryptionkey@0"
I20260806 09:15:48.523027 31350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 09:15:48.526304 31373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:48.527698 31350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:15:48.527885 31350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "dc91a6ae9a704e9cb20b135568f8558f"
format_stamp: "Formatted at 2026-08-06 09:15:48 on dist-test-slave-w4v5"
server_key: "4342565bdf1a68c4e979a6b67256fe08"
server_key_iv: "66618d4e0d9d6ac31be8968c9f698a62"
server_key_version: "encryptionkey@0"
I20260806 09:15:48.528079 31350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:48.541599 31350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:48.545894 31350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:48.546202 31350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:48.565706 31350 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:36321
I20260806 09:15:48.565716 31465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:36321 every 8 connection(s)
I20260806 09:15:48.567174 31350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:48.570045 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 31350
I20260806 09:15:48.570226 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:15:48.570735 3348 external_mini_cluster.cc:1428] Setting key 69687c71f53042eec3538c9c587cd422
I20260806 09:15:48.571040 31466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:48.585168 31466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f: Bootstrap starting.
Aug 06 09:15:48 dist-test-slave-w4v5 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:48.588320 31466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:48.589558 31466 log.cc:824] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:48.592587 31466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f: No bootstrap required, opened a new log
I20260806 09:15:48.595829 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:48.572730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.3.69.62:36321)
0806 09:15:48.573605 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:48.573619 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:48.573669 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:15:48.576113 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:48.576141 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:48.576617 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:48.577061 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:48.578772 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:48.580767 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:48.582503 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:48.583378 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:48.583586 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:48.588077 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:48.588084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:48.588104 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:48.588141 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:48.591495 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:48.592277 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:48.592284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:48.592291 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:48.592480 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:48.592925 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:48.592930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:48.592933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:48.593474 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:48.593812 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:48.594355 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 09:15:48.594782 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":367,"threads_started":1}
I20260806 09:15:48.600649 31466 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc91a6ae9a704e9cb20b135568f8558f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36321 } }
I20260806 09:15:48.601078 31466 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:48.601194 31466 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc91a6ae9a704e9cb20b135568f8558f, State: Initialized, Role: FOLLOWER
I20260806 09:15:48.601740 31466 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc91a6ae9a704e9cb20b135568f8558f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36321 } }
I20260806 09:15:48.602020 31466 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:48.602177 31466 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:48.602411 31466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:48.604734 31466 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc91a6ae9a704e9cb20b135568f8558f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36321 } }
I20260806 09:15:48.605257 31466 leader_election.cc:302] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc91a6ae9a704e9cb20b135568f8558f; no voters:
I20260806 09:15:48.605891 31466 leader_election.cc:288] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:15:48.606022 31474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:48.607864 31474 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [term 1 LEADER]: Becoming Leader. State: Replica: dc91a6ae9a704e9cb20b135568f8558f, State: Running, Role: LEADER
I20260806 09:15:48.608263 31466 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:48.608439 31474 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc91a6ae9a704e9cb20b135568f8558f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36321 } }
I20260806 09:15:48.611373 31476 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc91a6ae9a704e9cb20b135568f8558f. Latest consensus state: current_term: 1 leader_uuid: "dc91a6ae9a704e9cb20b135568f8558f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc91a6ae9a704e9cb20b135568f8558f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36321 } } }
I20260806 09:15:48.611778 31476 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:48.612524 31475 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc91a6ae9a704e9cb20b135568f8558f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc91a6ae9a704e9cb20b135568f8558f" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 36321 } } }
I20260806 09:15:48.612928 31475 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:48.616320 31490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:48.620657 31490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:48.629258 31490 catalog_manager.cc:1347] Generated new cluster ID: dc61bbf9f8084aeab70d96fb61924041
I20260806 09:15:48.629376 31490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:48.640316 31490 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:15:48.642095 31490 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:48.653108 31490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f: Generated new TSK 0
I20260806 09:15:48.653949 31490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.69.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
Entry for principal kudu/127.3.69.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab.
I20260806 09:15:48.735194 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36321
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:49.083022 31511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:49.083320 31511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:49.083416 31511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:49.090500 31511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:49.090606 31511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:49.090718 31511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:15:49.100703 31511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.1.keytab
--principal=kudu/127.3.69.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:49.101905 31511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:49.103180 31511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:49.116963 31519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:49.116932 31523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:49.117012 31520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:49.117697 31511 server_base.cc:1034] running on GCE node
I20260806 09:15:49.118371 31511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:49.119408 31511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:49.120637 31511 hybrid_clock.cc:648] HybridClock initialized: now 1786007749120568 us; error 62 us; skew 500 ppm
I20260806 09:15:49.120968 31511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:49 dist-test-slave-w4v5 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:49.124509 31511 init.cc:377] Logged in from keytab as kudu/127.3.69.1@KRBTEST.COM (short username kudu)
I20260806 09:15:49.126135 31511 webserver.cc:460] Webserver started at http://127.3.69.1:39045/ using document root <none> and password file <none>
I20260806 09:15:49.126878 31511 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:49.127034 31511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:49.127391 31511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:49.130410 31511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "4ef941f8f09b4668bcc219c03e4e3307"
format_stamp: "Formatted at 2026-08-06 09:15:49 on dist-test-slave-w4v5"
server_key: "ab285c04c60c6fa879cbb8281019918f"
server_key_iv: "207089720b081ed1a2f5616de28a6fa7"
server_key_version: "encryptionkey@0"
I20260806 09:15:49.131336 31511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "4ef941f8f09b4668bcc219c03e4e3307"
format_stamp: "Formatted at 2026-08-06 09:15:49 on dist-test-slave-w4v5"
server_key: "ab285c04c60c6fa879cbb8281019918f"
server_key_iv: "207089720b081ed1a2f5616de28a6fa7"
server_key_version: "encryptionkey@0"
I20260806 09:15:49.137019 31511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:15:49.140733 31532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:49.142122 31511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:15:49.142328 31511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "4ef941f8f09b4668bcc219c03e4e3307"
format_stamp: "Formatted at 2026-08-06 09:15:49 on dist-test-slave-w4v5"
server_key: "ab285c04c60c6fa879cbb8281019918f"
server_key_iv: "207089720b081ed1a2f5616de28a6fa7"
server_key_version: "encryptionkey@0"
I20260806 09:15:49.142549 31511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:49.166972 31511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:49.170939 31511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:49.171262 31511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:49.172246 31511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:49.173616 31511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:49.173700 31511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:49.173878 31511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:49.173942 31511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:49.209290 31511 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:35457
I20260806 09:15:49.209379 31706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:35457 every 8 connection(s)
I20260806 09:15:49.210630 31511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:15:49.216646 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 31511
I20260806 09:15:49.216881 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:15:49.217379 3348 external_mini_cluster.cc:1428] Setting key 8102762eec26458253e192023a33bba5
Aug 06 09:15:49 dist-test-slave-w4v5 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.1@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:49.233659 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:49.213547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:37061 (local address 127.3.69.62:36321)
0806 09:15:49.213995 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:49.214000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:49.215013 (+ 1013us) server_negotiation.cc:407] Connection header received
0806 09:15:49.216926 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:49.216933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:49.217026 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:49.217153 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:49.219025 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:49.220158 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:49.221770 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:49.222507 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:49.222660 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:49.226409 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:49.226418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:49.226426 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:49.226474 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:49.230141 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:49.230904 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:49.230912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:49.230918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:49.231114 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:49.231545 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:49.231552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:49.231558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:49.232016 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:49.232174 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:49.232903 (+ 729us) server_negotiation.cc:299] Negotiation successful
0806 09:15:49.233178 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":215,"threads_started":1}
I20260806 09:15:49.234354 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:49.214123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36321 (local address 127.3.69.1:37061)
0806 09:15:49.214839 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:49.214885 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:49.216616 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:49.217399 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:49.217407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:49.218056 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:49.218841 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:49.218865 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:49.220347 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:49.220351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:49.221607 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:49.221625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:49.222725 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:49.222730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:49.222926 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:49.223694 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:49.223720 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:49.226169 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:49.230292 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:49.230299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:49.230312 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:49.230791 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:49.231233 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:49.231237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:49.231239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:49.231439 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:49.232160 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:49.232166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:49.232507 (+ 341us) client_negotiation.cc:769] Sending connection context
0806 09:15:49.232752 (+ 245us) client_negotiation.cc:240] Negotiation successful
0806 09:15:49.233228 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":300,"threads_started":1}
I20260806 09:15:49.236013 31707 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36321
I20260806 09:15:49.236367 31707 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:49.237244 31707 heartbeater.cc:511] Master 127.3.69.62:36321 requested a full tablet report, sending...
I20260806 09:15:49.239519 31397 ts_manager.cc:194] Registered new tserver with Master: 4ef941f8f09b4668bcc219c03e4e3307 (127.3.69.1:35457)
I20260806 09:15:49.240974 31397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.1@KRBTEST.COM'} at 127.3.69.1:37061
WARNING: no policy specified for kudu/127.3.69.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
Entry for principal kudu/127.3.69.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab.
I20260806 09:15:49.294699 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36321
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:49.639882 31715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:49.640200 31715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:49.640374 31715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:49.647528 31715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:49.647662 31715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:49.647835 31715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:15:49.657372 31715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.2.keytab
--principal=kudu/127.3.69.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:49.658631 31715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:49.659955 31715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:49.672981 31724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:49.673071 31722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:49.673071 31726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:49.674206 31715 server_base.cc:1034] running on GCE node
I20260806 09:15:49.674904 31715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:49.675976 31715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:49.677205 31715 hybrid_clock.cc:648] HybridClock initialized: now 1786007749677146 us; error 41 us; skew 500 ppm
I20260806 09:15:49.677553 31715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:49 dist-test-slave-w4v5 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:49.681003 31715 init.cc:377] Logged in from keytab as kudu/127.3.69.2@KRBTEST.COM (short username kudu)
I20260806 09:15:49.682598 31715 webserver.cc:460] Webserver started at http://127.3.69.2:39185/ using document root <none> and password file <none>
I20260806 09:15:49.683418 31715 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:49.683552 31715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:49.683879 31715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:49.686745 31715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "604abe7e0d184893a4efdd92b200a550"
format_stamp: "Formatted at 2026-08-06 09:15:49 on dist-test-slave-w4v5"
server_key: "f0dfc66f326ae469213e698cffb53083"
server_key_iv: "69ee84fa8666af59ce2339ac9ddaa387"
server_key_version: "encryptionkey@0"
I20260806 09:15:49.687573 31715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "604abe7e0d184893a4efdd92b200a550"
format_stamp: "Formatted at 2026-08-06 09:15:49 on dist-test-slave-w4v5"
server_key: "f0dfc66f326ae469213e698cffb53083"
server_key_iv: "69ee84fa8666af59ce2339ac9ddaa387"
server_key_version: "encryptionkey@0"
I20260806 09:15:49.692675 31715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:15:49.696170 31739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:49.697671 31715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:15:49.697867 31715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "604abe7e0d184893a4efdd92b200a550"
format_stamp: "Formatted at 2026-08-06 09:15:49 on dist-test-slave-w4v5"
server_key: "f0dfc66f326ae469213e698cffb53083"
server_key_iv: "69ee84fa8666af59ce2339ac9ddaa387"
server_key_version: "encryptionkey@0"
I20260806 09:15:49.698086 31715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:49.717540 31715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:49.721505 31715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:49.721832 31715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:49.722898 31715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:49.724335 31715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:49.724412 31715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:49.724582 31715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:49.724637 31715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:49.759974 31715 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:41923
I20260806 09:15:49.760020 31926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:41923 every 8 connection(s)
I20260806 09:15:49.761286 31715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:15:49.767751 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 31715
I20260806 09:15:49.767983 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:15:49.768479 3348 external_mini_cluster.cc:1428] Setting key daf5ec451840ce430b1443a6d59f1aa9
Aug 06 09:15:49 dist-test-slave-w4v5 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007749, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.2@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:49.782881 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:49.763998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:33165 (local address 127.3.69.62:36321)
0806 09:15:49.764496 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:49.764501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:49.765366 (+ 865us) server_negotiation.cc:407] Connection header received
0806 09:15:49.767281 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:49.767288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:49.767384 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:49.767517 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:49.769425 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:49.770297 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:49.772074 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:49.772521 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:49.772661 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:49.776732 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:49.776738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:49.776741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:49.776774 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:49.779622 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:49.780557 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:49.780562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:49.780566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:49.780699 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:49.781189 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:49.781192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:49.781194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:49.781444 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:49.781630 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:49.782438 (+ 808us) server_negotiation.cc:299] Negotiation successful
0806 09:15:49.782611 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":288,"threads_started":1}
I20260806 09:15:49.784372 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:49.764487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36321 (local address 127.3.69.2:33165)
0806 09:15:49.765199 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:49.765237 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:49.766952 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:49.767739 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:49.767747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:49.768393 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:49.769228 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:49.769256 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:49.770541 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:49.770547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:49.771917 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:49.771930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:49.772760 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:49.772769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:49.773003 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:49.773694 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:49.773723 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:49.776405 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:49.779762 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:49.779773 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:49.779792 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:49.780416 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:49.780812 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:49.780820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:49.780826 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:49.781071 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:49.781585 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:49.781596 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:49.782032 (+ 436us) client_negotiation.cc:769] Sending connection context
0806 09:15:49.782345 (+ 313us) client_negotiation.cc:240] Negotiation successful
0806 09:15:49.782911 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":279,"threads_started":1}
I20260806 09:15:49.785868 31927 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36321
I20260806 09:15:49.786191 31927 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:49.786904 31927 heartbeater.cc:511] Master 127.3.69.62:36321 requested a full tablet report, sending...
I20260806 09:15:49.788410 31397 ts_manager.cc:194] Registered new tserver with Master: 604abe7e0d184893a4efdd92b200a550 (127.3.69.2:41923)
I20260806 09:15:49.789263 31397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.2@KRBTEST.COM'} at 127.3.69.2:33165
WARNING: no policy specified for kudu/127.3.69.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
Entry for principal kudu/127.3.69.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab.
I20260806 09:15:49.838352 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36321
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:43063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:50.182840 31938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:50.183174 31938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:50.183344 31938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:50.190577 31938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:50.190734 31938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:50.190900 31938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:15:50.200495 31938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:43063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.3.keytab
--principal=kudu/127.3.69.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:36321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:50.201846 31938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:50.203320 31938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:50.218436 31947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:50.218408 31948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:50.218402 31952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:50.219713 31938 server_base.cc:1034] running on GCE node
I20260806 09:15:50.220356 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:50.221331 31938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:50.222573 31938 hybrid_clock.cc:648] HybridClock initialized: now 1786007750222524 us; error 64 us; skew 500 ppm
I20260806 09:15:50.222914 31938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:50 dist-test-slave-w4v5 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:50.226289 31938 init.cc:377] Logged in from keytab as kudu/127.3.69.3@KRBTEST.COM (short username kudu)
I20260806 09:15:50.227885 31938 webserver.cc:460] Webserver started at http://127.3.69.3:39525/ using document root <none> and password file <none>
I20260806 09:15:50.228588 31938 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:50.228746 31938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:50.229072 31938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:50.232120 31938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "ab0d587e60274d03b3ab99696f198db5"
format_stamp: "Formatted at 2026-08-06 09:15:50 on dist-test-slave-w4v5"
server_key: "eb296e362ab2cc1d5b9c0e2eae1b468c"
server_key_iv: "3a1949fbabf0a2a37b60bba990bec474"
server_key_version: "encryptionkey@0"
I20260806 09:15:50.232947 31938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "ab0d587e60274d03b3ab99696f198db5"
format_stamp: "Formatted at 2026-08-06 09:15:50 on dist-test-slave-w4v5"
server_key: "eb296e362ab2cc1d5b9c0e2eae1b468c"
server_key_iv: "3a1949fbabf0a2a37b60bba990bec474"
server_key_version: "encryptionkey@0"
I20260806 09:15:50.238092 31938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:15:50.241534 31961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:50.242821 31938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:15:50.243011 31938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "ab0d587e60274d03b3ab99696f198db5"
format_stamp: "Formatted at 2026-08-06 09:15:50 on dist-test-slave-w4v5"
server_key: "eb296e362ab2cc1d5b9c0e2eae1b468c"
server_key_iv: "3a1949fbabf0a2a37b60bba990bec474"
server_key_version: "encryptionkey@0"
I20260806 09:15:50.243274 31938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:50.244635 31707 heartbeater.cc:503] Master 127.3.69.62:36321 was elected leader, sending a full tablet report...
I20260806 09:15:50.258081 31938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:50.262400 31938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:50.262710 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:50.263808 31938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:50.265146 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:50.265264 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:50.265427 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:50.265504 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:50.302891 31938 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:36221
I20260806 09:15:50.302927 32143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:36221 every 8 connection(s)
I20260806 09:15:50.304256 31938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:15:50.309582 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 31938
I20260806 09:15:50.309767 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:15:50.310341 3348 external_mini_cluster.cc:1428] Setting key c103441c0098e63771b6240484316ca6
Aug 06 09:15:50 dist-test-slave-w4v5 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007750, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.3@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:15:50.326021 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.306928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:54917 (local address 127.3.69.62:36321)
0806 09:15:50.307413 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.307417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.308419 (+ 1002us) server_negotiation.cc:407] Connection header received
0806 09:15:50.311113 (+ 2694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.311119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.311213 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.311336 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:50.313317 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.314084 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.315688 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.316103 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.316238 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.319751 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:50.319755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:50.319759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:50.319790 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:50.322702 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.323438 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.323442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.323444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.323567 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.324075 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.324081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.324084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.324380 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:50.324546 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.325426 (+ 880us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.325700 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":257,"threads_started":1}
I20260806 09:15:50.326922 32148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.307478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:36321 (local address 127.3.69.3:54917)
0806 09:15:50.308227 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:50.308314 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:50.310804 (+ 2490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:50.311601 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:50.311616 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:50.312316 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:50.313123 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.313143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.314228 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.314232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.315528 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.315540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.316370 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.316376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.316613 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.317255 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:15:50.317283 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:50.319510 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:50.322830 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:50.322837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:50.322848 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:50.323330 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:50.323682 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:15:50.323687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:15:50.323690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:15:50.323940 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:15:50.324516 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:50.324523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:50.324963 (+ 440us) client_negotiation.cc:769] Sending connection context
0806 09:15:50.325293 (+ 330us) client_negotiation.cc:240] Negotiation successful
0806 09:15:50.325759 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":329,"threads_started":1}
I20260806 09:15:50.328341 32144 heartbeater.cc:347] Connected to a master server at 127.3.69.62:36321
I20260806 09:15:50.328680 32144 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:50.329391 32144 heartbeater.cc:511] Master 127.3.69.62:36321 requested a full tablet report, sending...
I20260806 09:15:50.330739 31397 ts_manager.cc:194] Registered new tserver with Master: ab0d587e60274d03b3ab99696f198db5 (127.3.69.3:36221)
I20260806 09:15:50.331609 31397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.69.3@KRBTEST.COM'} at 127.3.69.3:54917
I20260806 09:15:50.337884 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:15:50.355005 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.342066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58546 (local address 127.3.69.62:36321)
0806 09:15:50.342215 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.342220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.342726 (+ 506us) server_negotiation.cc:407] Connection header received
0806 09:15:50.342852 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.342856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.342913 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.342992 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:50.344426 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.345227 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.346638 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.347134 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.347232 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.349975 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:50.349980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:50.349983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:50.350012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:50.352748 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.353404 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.353407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.353409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.353521 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.353867 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.353870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.353872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.354160 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:50.354345 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.354610 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.354782 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 09:15:50.358799 31397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:15:50.365892 31397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:15:50.391310 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.377906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:35457 (local address 127.0.0.1:37338)
0806 09:15:50.378952 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:50.379028 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:50.379267 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:50.381590 (+ 2323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:50.381603 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:50.381674 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:50.382326 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.382356 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.385506 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.385514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.388374 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.388415 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.389879 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.389886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.390112 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.390176 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 09:15:50.390314 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 09:15:50.390602 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":28,"thread_start_us":2609,"threads_started":1}
I20260806 09:15:50.391641 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.378076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.3.69.1:35457)
0806 09:15:50.380433 (+ 2357us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.380443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.380474 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:15:50.380597 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.380606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.380936 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.382636 (+ 1700us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:50.382992 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.385295 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.388568 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.389668 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.389830 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.390093 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.391082 (+ 989us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.391289 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2164,"thread_start_us":1996,"threads_started":1}
I20260806 09:15:50.392099 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.378681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.3.69.3:36221)
0806 09:15:50.380054 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.380065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.381245 (+ 1180us) server_negotiation.cc:407] Connection header received
0806 09:15:50.381351 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.381360 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.381625 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.381845 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:50.383124 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.385226 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.388970 (+ 3744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.390400 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.390558 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.390816 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.391525 (+ 709us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.391728 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":1023,"threads_started":1}
I20260806 09:15:50.392724 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.377981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:36221 (local address 127.0.0.1:56352)
0806 09:15:50.380952 (+ 2971us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:50.380974 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:50.381115 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:50.382216 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:50.382224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:50.382264 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:50.382837 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.382852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.385435 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.385443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.388796 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.388810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.391069 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.391097 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.391299 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.391332 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:15:50.392189 (+ 857us) client_negotiation.cc:240] Negotiation successful
0806 09:15:50.392321 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2847,"mutex_wait_us":718,"thread_start_us":564,"threads_started":1}
I20260806 09:15:50.395689 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.380584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:41923 (local address 127.0.0.1:50650)
0806 09:15:50.386655 (+ 6071us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:50.386673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:50.386812 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:50.387748 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:50.387753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:50.387779 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:50.388288 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.388300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.390716 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.390729 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.393196 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.393209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.394924 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.394931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.395124 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.395166 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 09:15:50.395260 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 09:15:50.395378 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5923,"thread_start_us":5710,"threads_started":1}
I20260806 09:15:50.395887 31607 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd8f0ef753b4856993ee56be16ad4a4 (DEFAULT_TABLE table=test-table [id=42cf13ec791e40afb75c589f96ce2ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:50.396620 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.381211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.3.69.2:41923)
0806 09:15:50.386160 (+ 4949us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.386173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.386943 (+ 770us) server_negotiation.cc:407] Connection header received
0806 09:15:50.387054 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.387061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.387316 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.387501 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:50.388478 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.390375 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.393356 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.394713 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.394864 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.395066 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.396110 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.396275 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4744,"thread_start_us":4601,"threads_started":1}
I20260806 09:15:50.396610 32044 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd8f0ef753b4856993ee56be16ad4a4 (DEFAULT_TABLE table=test-table [id=42cf13ec791e40afb75c589f96ce2ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:50.397312 31607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd8f0ef753b4856993ee56be16ad4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:50.397869 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd8f0ef753b4856993ee56be16ad4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:50.403254 31821 tablet_service.cc:1467] Processing CreateTablet for tablet 0bd8f0ef753b4856993ee56be16ad4a4 (DEFAULT_TABLE table=test-table [id=42cf13ec791e40afb75c589f96ce2ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:50.404654 31821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bd8f0ef753b4856993ee56be16ad4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:50.409716 32174 tablet_bootstrap.cc:492] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Bootstrap starting.
I20260806 09:15:50.413329 32174 tablet_bootstrap.cc:654] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:50.414476 32174 log.cc:824] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:50.417102 32174 tablet_bootstrap.cc:492] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: No bootstrap required, opened a new log
I20260806 09:15:50.417423 32174 ts_tablet_manager.cc:1397] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260806 09:15:50.418002 32175 tablet_bootstrap.cc:492] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: Bootstrap starting.
I20260806 09:15:50.421065 32178 tablet_bootstrap.cc:492] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Bootstrap starting.
I20260806 09:15:50.421917 32175 tablet_bootstrap.cc:654] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:50.423265 32175 log.cc:824] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:50.424460 32178 tablet_bootstrap.cc:654] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:50.425433 32178 log.cc:824] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:50.426415 32175 tablet_bootstrap.cc:492] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: No bootstrap required, opened a new log
I20260806 09:15:50.426728 32175 ts_tablet_manager.cc:1397] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260806 09:15:50.428043 32178 tablet_bootstrap.cc:492] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: No bootstrap required, opened a new log
I20260806 09:15:50.428306 32178 ts_tablet_manager.cc:1397] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 09:15:50.428449 32174 raft_consensus.cc:348] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.428892 32174 raft_consensus.cc:374] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:50.428993 32174 raft_consensus.cc:729] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef941f8f09b4668bcc219c03e4e3307, State: Initialized, Role: FOLLOWER
I20260806 09:15:50.429613 32174 consensus_queue.cc:260] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.431490 32174 ts_tablet_manager.cc:1428] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260806 09:15:50.437173 32175 raft_consensus.cc:348] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.437538 32175 raft_consensus.cc:374] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:50.437307 32178 raft_consensus.cc:348] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.437665 32175 raft_consensus.cc:729] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0d587e60274d03b3ab99696f198db5, State: Initialized, Role: FOLLOWER
I20260806 09:15:50.437711 32178 raft_consensus.cc:374] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:50.437819 32178 raft_consensus.cc:729] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604abe7e0d184893a4efdd92b200a550, State: Initialized, Role: FOLLOWER
I20260806 09:15:50.438283 32175 consensus_queue.cc:260] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.438380 32178 consensus_queue.cc:260] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.439520 32144 heartbeater.cc:503] Master 127.3.69.62:36321 was elected leader, sending a full tablet report...
I20260806 09:15:50.439527 31927 heartbeater.cc:503] Master 127.3.69.62:36321 was elected leader, sending a full tablet report...
I20260806 09:15:50.440145 32175 ts_tablet_manager.cc:1428] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260806 09:15:50.441212 32178 ts_tablet_manager.cc:1428] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260806 09:15:50.464109 31708 tablet.cc:2369] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:50.470062 32183 raft_consensus.cc:482] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:15:50.470350 32183 raft_consensus.cc:504] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.472112 32183 leader_election.cc:288] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab0d587e60274d03b3ab99696f198db5 (127.3.69.3:36221), 604abe7e0d184893a4efdd92b200a550 (127.3.69.2:41923)
I20260806 09:15:50.478860 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.472728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:39555 (local address 127.3.69.2:41923)
0806 09:15:50.472991 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.472997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.473461 (+ 464us) server_negotiation.cc:407] Connection header received
0806 09:15:50.473687 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.473692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.473753 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.473851 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:50.474455 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.475252 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.477089 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.477859 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.478015 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.478123 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.478461 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.478590 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:15:50.478902 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.472442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:42859 (local address 127.3.69.3:36221)
0806 09:15:50.472656 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.472662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.473493 (+ 831us) server_negotiation.cc:407] Connection header received
0806 09:15:50.473759 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.473763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.473824 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.473924 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:50.474588 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.475389 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.477127 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.477906 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.478060 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.478178 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.478514 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.478644 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:15:50.479216 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.472533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:36221 (local address 127.3.69.1:42859)
0806 09:15:50.473382 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:50.473397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:50.473602 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:50.473940 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:50.473944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:50.473995 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:50.474431 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.474441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.475539 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.475544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.476971 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.476984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.478142 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.478151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.478358 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.478382 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:15:50.478487 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 09:15:50.478609 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":38,"spinlock_wait_cycles":76800,"thread_start_us":541,"threads_started":1}
I20260806 09:15:50.479242 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.472380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:41923 (local address 127.3.69.1:39555)
0806 09:15:50.473345 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:50.473365 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:50.473536 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:50.473866 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:50.473870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:50.473894 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:50.474298 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.474310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.475378 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.475386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.476921 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:50.476936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.478099 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.478105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:50.478265 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.478325 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 09:15:50.478433 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 09:15:50.478562 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":42,"spinlock_wait_cycles":896,"thread_start_us":495,"threads_started":1}
I20260806 09:15:50.480281 31850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd8f0ef753b4856993ee56be16ad4a4" candidate_uuid: "4ef941f8f09b4668bcc219c03e4e3307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604abe7e0d184893a4efdd92b200a550" is_pre_election: true
I20260806 09:15:50.480232 32076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd8f0ef753b4856993ee56be16ad4a4" candidate_uuid: "4ef941f8f09b4668bcc219c03e4e3307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0d587e60274d03b3ab99696f198db5" is_pre_election: true
I20260806 09:15:50.480839 31850 raft_consensus.cc:2454] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ef941f8f09b4668bcc219c03e4e3307 in term 0.
I20260806 09:15:50.480890 32076 raft_consensus.cc:2454] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ef941f8f09b4668bcc219c03e4e3307 in term 0.
I20260806 09:15:50.481621 31536 leader_election.cc:302] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ef941f8f09b4668bcc219c03e4e3307, ab0d587e60274d03b3ab99696f198db5; no voters:
I20260806 09:15:50.482087 32183 raft_consensus.cc:2781] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:15:50.482251 32183 raft_consensus.cc:482] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:15:50.482369 32183 raft_consensus.cc:3037] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:50.484877 32183 raft_consensus.cc:504] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.485532 32183 leader_election.cc:288] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [CANDIDATE]: Term 1 election: Requested vote from peers ab0d587e60274d03b3ab99696f198db5 (127.3.69.3:36221), 604abe7e0d184893a4efdd92b200a550 (127.3.69.2:41923)
I20260806 09:15:50.485901 32076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd8f0ef753b4856993ee56be16ad4a4" candidate_uuid: "4ef941f8f09b4668bcc219c03e4e3307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0d587e60274d03b3ab99696f198db5"
I20260806 09:15:50.485985 31850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bd8f0ef753b4856993ee56be16ad4a4" candidate_uuid: "4ef941f8f09b4668bcc219c03e4e3307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604abe7e0d184893a4efdd92b200a550"
I20260806 09:15:50.486093 32076 raft_consensus.cc:3037] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:50.486217 31850 raft_consensus.cc:3037] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:50.488442 32076 raft_consensus.cc:2454] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ef941f8f09b4668bcc219c03e4e3307 in term 1.
I20260806 09:15:50.488602 31850 raft_consensus.cc:2454] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ef941f8f09b4668bcc219c03e4e3307 in term 1.
I20260806 09:15:50.488833 31536 leader_election.cc:302] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ef941f8f09b4668bcc219c03e4e3307, ab0d587e60274d03b3ab99696f198db5; no voters:
I20260806 09:15:50.489256 32183 raft_consensus.cc:2781] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:50.489707 32183 raft_consensus.cc:686] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef941f8f09b4668bcc219c03e4e3307, State: Running, Role: LEADER
I20260806 09:15:50.490182 32183 consensus_queue.cc:237] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } }
I20260806 09:15:50.494539 31397 catalog_manager.cc:5495] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ef941f8f09b4668bcc219c03e4e3307 (127.3.69.1). New cstate: current_term: 1 leader_uuid: "4ef941f8f09b4668bcc219c03e4e3307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef941f8f09b4668bcc219c03e4e3307" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 35457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 } health_report { overall_health: UNKNOWN } } }
W20260806 09:15:50.513482 31928 tablet.cc:2369] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:50.516443 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.511170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.3.69.1:35457)
0806 09:15:50.511348 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.511352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.511511 (+ 159us) server_negotiation.cc:407] Connection header received
0806 09:15:50.511678 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.511682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.511767 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.511938 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:15:50.512308 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.512993 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.514502 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.514914 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.515011 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.515506 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:15:50.515993 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:15:50.516099 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.516183 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.516247 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 09:15:50.524125 32076 raft_consensus.cc:1261] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 1 FOLLOWER]: Refusing update from remote peer 4ef941f8f09b4668bcc219c03e4e3307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:50.524166 31850 raft_consensus.cc:1261] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 1 FOLLOWER]: Refusing update from remote peer 4ef941f8f09b4668bcc219c03e4e3307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:50.525084 32188 consensus_queue.cc:1035] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604abe7e0d184893a4efdd92b200a550" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 41923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:50.525425 32183 consensus_queue.cc:1035] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0d587e60274d03b3ab99696f198db5" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 36221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:50.537122 32193 mvcc.cc:205] Tried to move back new op lower bound from 7315487746137546752 to 7315487746011869184. Current Snapshot: MvccSnapshot[applied={T|T < 7315487746137546752}]
I20260806 09:15:50.552198 31395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:50.552507 31395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:50.556149 31395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc91a6ae9a704e9cb20b135568f8558f: Sending DeleteTablet for 3 replicas of tablet 0bd8f0ef753b4856993ee56be16ad4a4
I20260806 09:15:50.557377 31607 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd8f0ef753b4856993ee56be16ad4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:50 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:37338
I20260806 09:15:50.557410 32044 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd8f0ef753b4856993ee56be16ad4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:50 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:56352
I20260806 09:15:50.558012 31821 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bd8f0ef753b4856993ee56be16ad4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:50 UTC) from {username='kudu', principal='kudu/127.3.69.62@KRBTEST.COM'} at 127.0.0.1:50650
I20260806 09:15:50.558454 32209 tablet_replica.cc:330] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: stopping tablet replica
I20260806 09:15:50.558846 32209 raft_consensus.cc:2229] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:15:50.559496 32209 raft_consensus.cc:2258] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:50.559639 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 31511
I20260806 09:15:50.560508 32212 tablet_replica.cc:330] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: stopping tablet replica
I20260806 09:15:50.560880 32212 raft_consensus.cc:2229] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:50.561405 32212 raft_consensus.cc:2258] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:50.563597 32214 tablet_replica.cc:330] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: stopping tablet replica
I20260806 09:15:50.564021 32214 raft_consensus.cc:2229] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:50.564276 32209 ts_tablet_manager.cc:1905] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:50.564612 32214 raft_consensus.cc:2258] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:50.566473 32212 ts_tablet_manager.cc:1905] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:50.568279 32214 ts_tablet_manager.cc:1905] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:50.571499 32209 ts_tablet_manager.cc:1918] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:50.571736 32209 log.cc:1196] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/0bd8f0ef753b4856993ee56be16ad4a4
I20260806 09:15:50.572268 32209 ts_tablet_manager.cc:1939] T 0bd8f0ef753b4856993ee56be16ad4a4 P 4ef941f8f09b4668bcc219c03e4e3307: Deleting consensus metadata
Aug 06 09:15:50 dist-test-slave-w4v5 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.1@KRBTEST.COM
I20260806 09:15:50.574026 31376 catalog_manager.cc:4841] TS 4ef941f8f09b4668bcc219c03e4e3307 (127.3.69.1:35457): tablet 0bd8f0ef753b4856993ee56be16ad4a4 (table test-table [id=42cf13ec791e40afb75c589f96ce2ef4]) successfully deleted
I20260806 09:15:50.575789 32212 ts_tablet_manager.cc:1918] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:50.575965 32212 log.cc:1196] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/0bd8f0ef753b4856993ee56be16ad4a4
I20260806 09:15:50.576524 32212 ts_tablet_manager.cc:1939] T 0bd8f0ef753b4856993ee56be16ad4a4 P ab0d587e60274d03b3ab99696f198db5: Deleting consensus metadata
I20260806 09:15:50.578378 31378 catalog_manager.cc:4841] TS ab0d587e60274d03b3ab99696f198db5 (127.3.69.3:36221): tablet 0bd8f0ef753b4856993ee56be16ad4a4 (table test-table [id=42cf13ec791e40afb75c589f96ce2ef4]) successfully deleted
I20260806 09:15:50.582188 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.561844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.3.69.1:35457)
0806 09:15:50.562017 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.562023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.562041 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:15:50.562100 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.562107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.562189 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.562287 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:50.564309 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.565511 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.567450 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.568170 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.568276 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.574481 (+ 6205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:50.574497 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:50.574515 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:50.574573 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:50.578602 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.579498 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.579507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.579510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.579682 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.580169 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.580176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.580179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.580634 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:50.580929 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.581548 (+ 619us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.581821 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 09:15:50.583935 31702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:50.585462 32214 ts_tablet_manager.cc:1918] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:50.585754 32214 log.cc:1196] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/0bd8f0ef753b4856993ee56be16ad4a4
I20260806 09:15:50.586493 32214 ts_tablet_manager.cc:1939] T 0bd8f0ef753b4856993ee56be16ad4a4 P 604abe7e0d184893a4efdd92b200a550: Deleting consensus metadata
I20260806 09:15:50.588850 31375 catalog_manager.cc:4841] TS 604abe7e0d184893a4efdd92b200a550 (127.3.69.2:41923): tablet 0bd8f0ef753b4856993ee56be16ad4a4 (table test-table [id=42cf13ec791e40afb75c589f96ce2ef4]) successfully deleted
I20260806 09:15:50.687151 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 31511
I20260806 09:15:50.712298 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 31715
Aug 06 09:15:50 dist-test-slave-w4v5 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.2@KRBTEST.COM
I20260806 09:15:50.728551 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.713140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.3.69.2:41923)
0806 09:15:50.713344 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.713348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.713367 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:15:50.713527 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.713530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.713595 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.713671 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:50.715012 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.715919 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.717351 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.717738 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.717834 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.722673 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:50.722686 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:50.722712 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:50.722750 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:50.725728 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.726413 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.726421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.726423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.726545 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.726889 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.726893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.726895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.727455 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:50.727683 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.728018 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.728181 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:15:50.729200 31919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:50.824577 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 31715
I20260806 09:15:50.853124 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 31938
Aug 06 09:15:50 dist-test-slave-w4v5 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.3@KRBTEST.COM
I20260806 09:15:50.868374 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:50.854036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.3.69.3:36221)
0806 09:15:50.854244 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:50.854249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:50.854268 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:15:50.854431 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:50.854435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:50.854505 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:50.854596 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:50.856129 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.856859 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.858103 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:50.858499 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:50.858602 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:50.862943 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:50.862958 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:50.862975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:50.863008 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:50.865867 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.866528 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.866536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.866538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.866677 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:50.867024 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:50.867027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:50.867029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:50.867475 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:50.867713 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:50.867945 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 09:15:50.868110 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:15:50.869005 32137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:50.958909 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 31938
I20260806 09:15:50.984768 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 31350
I20260806 09:15:50.985968 31456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:51.097846 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 31350
2026-08-06T09:15:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4634 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T09:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:15:51Z Disabled control of system clock
I20260806 09:15:51.202576 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40833
--webserver_interface=127.3.69.62
--webserver_port=0
--builtin_ntp_servers=127.3.69.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:40833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:15:51.544790 32231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:51.545058 32231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:51.545140 32231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:51.552575 32231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:51.552687 32231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:51.552727 32231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:51.552778 32231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:51.552860 32231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:51.561934 32231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:40833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:51.563122 32231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:51.564836 32231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:51.577339 32239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:51.577344 32240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:51.577384 32242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:51.578419 32231 server_base.cc:1034] running on GCE node
I20260806 09:15:51.579259 32231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:51.580647 32231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:51.581882 32231 hybrid_clock.cc:648] HybridClock initialized: now 1786007751581844 us; error 50 us; skew 500 ppm
I20260806 09:15:51.582244 32231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:51.584538 32231 webserver.cc:460] Webserver started at http://127.3.69.62:42295/ using document root <none> and password file <none>
I20260806 09:15:51.585222 32231 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:51.585357 32231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:51.585772 32231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:51.588762 32231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "a75b9fee4e184feda020749c25dc9a14"
format_stamp: "Formatted at 2026-08-06 09:15:51 on dist-test-slave-w4v5"
server_key: "1f1ee4dc08fe58978e8a67cf030b4d88"
server_key_iv: "d2436144c16aef1fda8745ce58ee2ef8"
server_key_version: "encryptionkey@0"
I20260806 09:15:51.589745 32231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "a75b9fee4e184feda020749c25dc9a14"
format_stamp: "Formatted at 2026-08-06 09:15:51 on dist-test-slave-w4v5"
server_key: "1f1ee4dc08fe58978e8a67cf030b4d88"
server_key_iv: "d2436144c16aef1fda8745ce58ee2ef8"
server_key_version: "encryptionkey@0"
I20260806 09:15:51.595757 32231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260806 09:15:51.599007 32250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:51.600387 32231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:15:51.600589 32231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "a75b9fee4e184feda020749c25dc9a14"
format_stamp: "Formatted at 2026-08-06 09:15:51 on dist-test-slave-w4v5"
server_key: "1f1ee4dc08fe58978e8a67cf030b4d88"
server_key_iv: "d2436144c16aef1fda8745ce58ee2ef8"
server_key_version: "encryptionkey@0"
I20260806 09:15:51.600811 32231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:51.621523 32231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:51.622377 32231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:51.622689 32231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:51.642168 32231 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:40833
I20260806 09:15:51.642192 32329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:40833 every 8 connection(s)
I20260806 09:15:51.643654 32231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:51.647482 32331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:51.650889 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 32231
I20260806 09:15:51.651172 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:15:51.651645 3348 external_mini_cluster.cc:1428] Setting key 3534cef622d472bda4a04de5292167a2
I20260806 09:15:51.659976 32331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14: Bootstrap starting.
I20260806 09:15:51.663317 32331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:51.664485 32331 log.cc:824] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:51.665418 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:51.653651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.3.69.62:40833)
0806 09:15:51.654762 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:51.654773 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:51.654815 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:15:51.656558 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:51.656581 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:51.657042 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:51.657448 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:51.658052 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:51.659370 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:51.660825 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:51.661525 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:51.661740 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:51.662984 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:51.662991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:51.663005 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:51.663653 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:51.663752 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:51.663778 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:51.663911 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:51.664154 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 09:15:51.664434 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":502,"threads_started":1}
I20260806 09:15:51.667455 32331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14: No bootstrap required, opened a new log
I20260806 09:15:51.675222 32331 raft_consensus.cc:348] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b9fee4e184feda020749c25dc9a14" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40833 } }
I20260806 09:15:51.675598 32331 raft_consensus.cc:374] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:51.675732 32331 raft_consensus.cc:729] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75b9fee4e184feda020749c25dc9a14, State: Initialized, Role: FOLLOWER
I20260806 09:15:51.676286 32331 consensus_queue.cc:260] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b9fee4e184feda020749c25dc9a14" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40833 } }
I20260806 09:15:51.676529 32331 raft_consensus.cc:388] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:51.676684 32331 raft_consensus.cc:482] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:51.676826 32331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:51.679411 32331 raft_consensus.cc:504] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b9fee4e184feda020749c25dc9a14" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40833 } }
I20260806 09:15:51.679885 32331 leader_election.cc:302] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a75b9fee4e184feda020749c25dc9a14; no voters:
I20260806 09:15:51.680430 32331 leader_election.cc:288] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:15:51.680596 32339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:51.681380 32339 raft_consensus.cc:686] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [term 1 LEADER]: Becoming Leader. State: Replica: a75b9fee4e184feda020749c25dc9a14, State: Running, Role: LEADER
I20260806 09:15:51.681926 32339 consensus_queue.cc:237] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b9fee4e184feda020749c25dc9a14" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40833 } }
I20260806 09:15:51.682605 32331 sys_catalog.cc:564] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:51.685288 32340 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a75b9fee4e184feda020749c25dc9a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b9fee4e184feda020749c25dc9a14" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40833 } } }
I20260806 09:15:51.686620 32340 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:51.685570 32341 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a75b9fee4e184feda020749c25dc9a14. Latest consensus state: current_term: 1 leader_uuid: "a75b9fee4e184feda020749c25dc9a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75b9fee4e184feda020749c25dc9a14" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40833 } } }
I20260806 09:15:51.687232 32341 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14 [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:51.688762 32362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:51.693434 32362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:51.702186 32362 catalog_manager.cc:1347] Generated new cluster ID: a2d60617acce4e94912e8ca724a6f661
I20260806 09:15:51.702304 32362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:51.711911 32362 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:15:51.713174 32362 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:51.723663 32362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14: Generated new TSK 0
I20260806 09:15:51.724444 32362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:15:51.738716 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.1:0
--local_ip_for_outbound_sockets=127.3.69.1
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40833
--builtin_ntp_servers=127.3.69.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:15:52.088512 32377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:52.088825 32377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:52.088994 32377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:52.096220 32377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:52.096354 32377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:52.096524 32377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.1
I20260806 09:15:52.105690 32377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.69.1
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.1
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:52.106958 32377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:52.108717 32377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:52.122016 32385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:52.122102 32386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:52.122129 32389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:52.123031 32377 server_base.cc:1034] running on GCE node
I20260806 09:15:52.123685 32377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:52.124670 32377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:52.125906 32377 hybrid_clock.cc:648] HybridClock initialized: now 1786007752125872 us; error 58 us; skew 500 ppm
I20260806 09:15:52.126224 32377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:52.128456 32377 webserver.cc:460] Webserver started at http://127.3.69.1:34259/ using document root <none> and password file <none>
I20260806 09:15:52.129130 32377 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:52.129273 32377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:52.129596 32377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:52.132527 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/data/instance:
uuid: "0feab77147654933a6bce9853d1e0073"
format_stamp: "Formatted at 2026-08-06 09:15:52 on dist-test-slave-w4v5"
server_key: "70f89a6b49de0f8048114e20d71ab7fe"
server_key_iv: "c8efef31d185cf444fa16f0396f91de0"
server_key_version: "encryptionkey@0"
I20260806 09:15:52.133315 32377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance:
uuid: "0feab77147654933a6bce9853d1e0073"
format_stamp: "Formatted at 2026-08-06 09:15:52 on dist-test-slave-w4v5"
server_key: "70f89a6b49de0f8048114e20d71ab7fe"
server_key_iv: "c8efef31d185cf444fa16f0396f91de0"
server_key_version: "encryptionkey@0"
I20260806 09:15:52.138244 32377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:15:52.141674 32398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:52.142931 32377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:15:52.143174 32377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/wal
uuid: "0feab77147654933a6bce9853d1e0073"
format_stamp: "Formatted at 2026-08-06 09:15:52 on dist-test-slave-w4v5"
server_key: "70f89a6b49de0f8048114e20d71ab7fe"
server_key_iv: "c8efef31d185cf444fa16f0396f91de0"
server_key_version: "encryptionkey@0"
I20260806 09:15:52.143399 32377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:52.162384 32377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:52.163271 32377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:52.163553 32377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:52.164613 32377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:52.166348 32377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:52.166440 32377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:52.166519 32377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:52.166575 32377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:52.201669 32377 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.1:46197
I20260806 09:15:52.201694 32581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.1:46197 every 8 connection(s)
I20260806 09:15:52.202994 32377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/data/info.pb
I20260806 09:15:52.212616 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 32377
I20260806 09:15:52.212796 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/wal/instance
I20260806 09:15:52.213376 3348 external_mini_cluster.cc:1428] Setting key 5ad2b04163f425aa623b640afd309dd4
I20260806 09:15:52.216151 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:52.205787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:49907 (local address 127.3.69.62:40833)
0806 09:15:52.206253 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:52.206259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:52.207601 (+ 1342us) server_negotiation.cc:407] Connection header received
0806 09:15:52.209339 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:52.209344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:52.209410 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:52.209525 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:52.210545 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:52.211350 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:52.213252 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:52.213728 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:52.213833 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:52.215283 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:52.215288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:52.215291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:52.215307 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:52.215351 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:52.215363 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:52.215462 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:52.215689 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 09:15:52.215791 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260806 09:15:52.216368 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.2:0
--local_ip_for_outbound_sockets=127.3.69.2
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40833
--builtin_ntp_servers=127.3.69.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:15:52.217272 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:52.206453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40833 (local address 127.3.69.1:49907)
0806 09:15:52.207433 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:52.207470 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:52.209075 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:52.209729 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:52.209736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:52.209815 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:52.210367 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:52.210382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:52.211490 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:52.211494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:52.213107 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:52.213122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:52.213982 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:52.213988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:52.214182 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:52.214877 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:52.214881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:52.214938 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:52.215041 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:52.215464 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:52.215478 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:52.215488 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:15:52.215678 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 09:15:52.216064 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":290,"threads_started":1}
I20260806 09:15:52.218750 32582 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40833
I20260806 09:15:52.219107 32582 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:52.219831 32582 heartbeater.cc:511] Master 127.3.69.62:40833 requested a full tablet report, sending...
I20260806 09:15:52.221820 32278 ts_manager.cc:194] Registered new tserver with Master: 0feab77147654933a6bce9853d1e0073 (127.3.69.1:46197)
I20260806 09:15:52.223542 32278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.1:49907
W20260806 09:15:52.560883 32589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:52.561139 32589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:52.561224 32589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:52.568116 32589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:52.568208 32589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:52.568310 32589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.2
I20260806 09:15:52.577394 32589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.69.2
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.2
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:52.578649 32589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:52.580818 32589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:52.596554 32597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:52.596650 32595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:52.596748 32601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:52.598655 32589 server_base.cc:1034] running on GCE node
I20260806 09:15:52.599424 32589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:52.600354 32589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:52.601569 32589 hybrid_clock.cc:648] HybridClock initialized: now 1786007752601516 us; error 39 us; skew 500 ppm
I20260806 09:15:52.601925 32589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:52.604457 32589 webserver.cc:460] Webserver started at http://127.3.69.2:33233/ using document root <none> and password file <none>
I20260806 09:15:52.605094 32589 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:52.605182 32589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:52.605535 32589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:52.608428 32589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/data/instance:
uuid: "e587366475654eb6aaf01aa73a0af39a"
format_stamp: "Formatted at 2026-08-06 09:15:52 on dist-test-slave-w4v5"
server_key: "81ff2097a72dbefa38b52fd3a9684b35"
server_key_iv: "87daf89342a039ed90e9eb985334567b"
server_key_version: "encryptionkey@0"
I20260806 09:15:52.609158 32589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance:
uuid: "e587366475654eb6aaf01aa73a0af39a"
format_stamp: "Formatted at 2026-08-06 09:15:52 on dist-test-slave-w4v5"
server_key: "81ff2097a72dbefa38b52fd3a9684b35"
server_key_iv: "87daf89342a039ed90e9eb985334567b"
server_key_version: "encryptionkey@0"
I20260806 09:15:52.614159 32589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:15:52.617513 32608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:52.618896 32589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:15:52.619024 32589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/wal
uuid: "e587366475654eb6aaf01aa73a0af39a"
format_stamp: "Formatted at 2026-08-06 09:15:52 on dist-test-slave-w4v5"
server_key: "81ff2097a72dbefa38b52fd3a9684b35"
server_key_iv: "87daf89342a039ed90e9eb985334567b"
server_key_version: "encryptionkey@0"
I20260806 09:15:52.619289 32589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:52.659281 32589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:52.660048 32589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:52.660367 32589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:52.661372 32589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:52.662830 32589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:52.662919 32589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:52.662993 32589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:52.663049 32589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:52.697177 32589 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.2:38887
I20260806 09:15:52.697232 307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.2:38887 every 8 connection(s)
I20260806 09:15:52.698511 32589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/data/info.pb
I20260806 09:15:52.701206 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 32589
I20260806 09:15:52.701385 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/wal/instance
I20260806 09:15:52.701946 3348 external_mini_cluster.cc:1428] Setting key abd50abd8d0794d0129f05f98342611f
I20260806 09:15:52.705538 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.69.3:0
--local_ip_for_outbound_sockets=127.3.69.3
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40833
--builtin_ntp_servers=127.3.69.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:15:52.715672 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:52.701228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.2:58211 (local address 127.3.69.62:40833)
0806 09:15:52.701454 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:52.701461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:52.703778 (+ 2317us) server_negotiation.cc:407] Connection header received
0806 09:15:52.706666 (+ 2888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:52.706702 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:52.706820 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:52.706936 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:52.708636 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:52.710040 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:52.712339 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:52.712905 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:52.713030 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:52.714686 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:52.714694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:52.714699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:52.714724 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:52.714770 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:52.714789 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:52.714887 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:52.715147 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 09:15:52.715272 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:15:52.716512 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:52.702034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40833 (local address 127.3.69.2:58211)
0806 09:15:52.703613 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:52.703668 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:52.706286 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:52.707328 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:52.707343 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:52.707506 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:52.708421 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:52.708449 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:52.710288 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:52.710296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:52.712191 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:52.712204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:52.713316 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:52.713327 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:52.713621 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:52.714352 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:52.714354 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:52.714393 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:52.714472 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:52.714890 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:52.714899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:52.714905 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:15:52.715108 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 09:15:52.715411 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":679,"threads_started":1}
I20260806 09:15:52.717976 308 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40833
I20260806 09:15:52.718312 308 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:52.719272 308 heartbeater.cc:511] Master 127.3.69.62:40833 requested a full tablet report, sending...
I20260806 09:15:52.720907 32278 ts_manager.cc:194] Registered new tserver with Master: e587366475654eb6aaf01aa73a0af39a (127.3.69.2:38887)
I20260806 09:15:52.721767 32278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.2:58211
W20260806 09:15:53.057367 313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:53.057708 313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:53.057883 313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:53.064905 313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:53.065040 313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:53.065228 313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.69.3
I20260806 09:15:53.074644 313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.69.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.69.3
--webserver_port=0
--tserver_master_addrs=127.3.69.62:40833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.69.3
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:53.075934 313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:53.077737 313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:53.091158 328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:53.091256 323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:53.091229 325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:53.092214 313 server_base.cc:1034] running on GCE node
I20260806 09:15:53.092865 313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:53.093842 313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:53.095129 313 hybrid_clock.cc:648] HybridClock initialized: now 1786007753095043 us; error 56 us; skew 500 ppm
I20260806 09:15:53.095480 313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:15:53.100731 313 webserver.cc:460] Webserver started at http://127.3.69.3:45897/ using document root <none> and password file <none>
I20260806 09:15:53.101450 313 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:53.101588 313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:53.101926 313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:53.105096 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/data/instance:
uuid: "c298d56975a34558b70663c38dcb8ee1"
format_stamp: "Formatted at 2026-08-06 09:15:53 on dist-test-slave-w4v5"
server_key: "c72d00ece7e017eb20572cb7de6310ec"
server_key_iv: "e8f52f58da838d6c54c493d8efa12b1f"
server_key_version: "encryptionkey@0"
I20260806 09:15:53.105962 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance:
uuid: "c298d56975a34558b70663c38dcb8ee1"
format_stamp: "Formatted at 2026-08-06 09:15:53 on dist-test-slave-w4v5"
server_key: "c72d00ece7e017eb20572cb7de6310ec"
server_key_iv: "e8f52f58da838d6c54c493d8efa12b1f"
server_key_version: "encryptionkey@0"
I20260806 09:15:53.111064 313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 09:15:53.114496 344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:53.115823 313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:15:53.116004 313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/wal
uuid: "c298d56975a34558b70663c38dcb8ee1"
format_stamp: "Formatted at 2026-08-06 09:15:53 on dist-test-slave-w4v5"
server_key: "c72d00ece7e017eb20572cb7de6310ec"
server_key_iv: "e8f52f58da838d6c54c493d8efa12b1f"
server_key_version: "encryptionkey@0"
I20260806 09:15:53.116241 313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:53.127902 313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:53.128660 313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:53.128937 313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:53.129912 313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:15:53.131651 313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:15:53.131778 313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:53.131919 313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:15:53.132009 313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:53.166877 313 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.3:41211
I20260806 09:15:53.166919 535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.3:41211 every 8 connection(s)
I20260806 09:15:53.168252 313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/data/info.pb
I20260806 09:15:53.171826 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 313
I20260806 09:15:53.172008 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/wal/instance
I20260806 09:15:53.172468 3348 external_mini_cluster.cc:1428] Setting key ed072ac6cdca3dc10a7d069df4493ac6
I20260806 09:15:53.181160 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.170969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.3:53205 (local address 127.3.69.62:40833)
0806 09:15:53.171167 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.171172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.172745 (+ 1573us) server_negotiation.cc:407] Connection header received
0806 09:15:53.175216 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.175224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.175318 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.175403 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:53.176427 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.177273 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.178731 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.179235 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.179332 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.180517 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:53.180521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:53.180524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:53.180541 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:53.180573 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:53.180584 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:53.180655 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.180853 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.180927 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 09:15:53.182082 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.171508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:40833 (local address 127.3.69.3:53205)
0806 09:15:53.172556 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.172607 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:53.174895 (+ 2288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:53.175647 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:53.175654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:53.175742 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:15:53.176288 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.176305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.177413 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.177417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.178592 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.178603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.179419 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.179424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.179612 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.180239 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:15:53.180242 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:15:53.180281 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:15:53.180348 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:15:53.180677 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:15:53.180685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:15:53.180690 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 09:15:53.180827 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 09:15:53.181084 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":358,"threads_started":1}
I20260806 09:15:53.183457 538 heartbeater.cc:347] Connected to a master server at 127.3.69.62:40833
I20260806 09:15:53.183750 538 heartbeater.cc:464] Registering TS with master...
I20260806 09:15:53.184435 538 heartbeater.cc:511] Master 127.3.69.62:40833 requested a full tablet report, sending...
I20260806 09:15:53.185799 32278 ts_manager.cc:194] Registered new tserver with Master: c298d56975a34558b70663c38dcb8ee1 (127.3.69.3:41211)
I20260806 09:15:53.186662 32278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.69.3:53205
I20260806 09:15:53.187860 3348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:15:53.197745 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.192044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.3.69.62:40833)
0806 09:15:53.192291 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.192301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.192694 (+ 393us) server_negotiation.cc:407] Connection header received
0806 09:15:53.192815 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.192819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.192886 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.192963 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:53.193379 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.194027 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.195426 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.195851 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.195961 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.197051 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:53.197055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:53.197059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:53.197074 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:53.197106 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:53.197117 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:53.197191 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.197331 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.197402 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 09:15:53.201756 32278 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:15:53.209054 32278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:15:53.226395 32582 heartbeater.cc:503] Master 127.3.69.62:40833 was elected leader, sending a full tablet report...
I20260806 09:15:53.246834 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.221694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46197 (local address 127.0.0.1:52906)
0806 09:15:53.235953 (+ 14259us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.235978 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:53.236131 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:53.238209 (+ 2078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:53.238220 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:53.238266 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:53.238703 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.238715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.241318 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.241324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.243805 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.243820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.245761 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.245767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.245955 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.246012 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 09:15:53.246181 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 09:15:53.246372 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14078,"mutex_wait_us":55,"thread_start_us":12726,"threads_started":1}
I20260806 09:15:53.246804 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.221913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.3.69.1:46197)
0806 09:15:53.237052 (+ 15139us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.237061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.237092 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:15:53.237216 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.237224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.237505 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.237729 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:53.238893 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.241174 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.244069 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.245567 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.245730 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.246017 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.246266 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.246425 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14960,"thread_start_us":16001,"threads_started":1}
I20260806 09:15:53.247344 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.221267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:38887 (local address 127.0.0.1:55272)
0806 09:15:53.234842 (+ 13575us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.234898 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:53.235107 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:53.239703 (+ 4596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:53.239708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:53.239738 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:53.240225 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.240238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.242108 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.242113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.243978 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.243991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.246195 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.246202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.246378 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.246412 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:15:53.246504 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 09:15:53.246639 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13312,"mutex_wait_us":24,"thread_start_us":14971,"threads_started":1}
I20260806 09:15:53.248181 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.224054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41211 (local address 127.0.0.1:39400)
0806 09:15:53.239880 (+ 15826us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.239901 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:53.240030 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:53.240641 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:53.240646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:53.240669 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:53.241033 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.241065 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.242931 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.242942 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.245252 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.245302 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.247430 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.247440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.247621 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.247652 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 09:15:53.247744 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 09:15:53.247861 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15702,"mutex_wait_us":2,"thread_start_us":15553,"threads_started":1}
I20260806 09:15:53.248291 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.221456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.3.69.2:38887)
0806 09:15:53.238769 (+ 17313us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.238780 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.238814 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:15:53.238933 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.238943 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.239420 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.240244 (+ 824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:53.240476 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.241960 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.244184 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.245678 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.245830 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.246054 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.247768 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.247931 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17097,"thread_start_us":17615,"threads_started":1}
I20260806 09:15:53.249387 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.224835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.3.69.3:41211)
0806 09:15:53.239435 (+ 14600us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.239444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.240026 (+ 582us) server_negotiation.cc:407] Connection header received
0806 09:15:53.240208 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.240216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.240433 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.240642 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:53.241295 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.242770 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.245490 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.246867 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.247004 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.247287 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.248793 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.248987 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14420,"thread_start_us":14229,"threads_started":1}
I20260806 09:15:53.251209 32473 tablet_service.cc:1467] Processing CreateTablet for tablet c2f356474dcf493284db105095c054bc (DEFAULT_TABLE table=test-table [id=457dc428bcbe4a23b631b4f443ce1cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:53.251839 32674 tablet_service.cc:1467] Processing CreateTablet for tablet c2f356474dcf493284db105095c054bc (DEFAULT_TABLE table=test-table [id=457dc428bcbe4a23b631b4f443ce1cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:53.252841 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f356474dcf493284db105095c054bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:53.253225 32674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f356474dcf493284db105095c054bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:53.253232 425 tablet_service.cc:1467] Processing CreateTablet for tablet c2f356474dcf493284db105095c054bc (DEFAULT_TABLE table=test-table [id=457dc428bcbe4a23b631b4f443ce1cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:15:53.254601 425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f356474dcf493284db105095c054bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:53.269232 570 tablet_bootstrap.cc:492] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Bootstrap starting.
I20260806 09:15:53.270253 573 tablet_bootstrap.cc:492] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Bootstrap starting.
I20260806 09:15:53.274222 573 tablet_bootstrap.cc:654] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:53.274240 570 tablet_bootstrap.cc:654] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:53.275305 572 tablet_bootstrap.cc:492] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Bootstrap starting.
I20260806 09:15:53.275585 573 log.cc:824] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:53.275585 570 log.cc:824] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:53.278546 572 tablet_bootstrap.cc:654] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:53.279806 572 log.cc:824] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:53.282003 573 tablet_bootstrap.cc:492] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: No bootstrap required, opened a new log
I20260806 09:15:53.281996 570 tablet_bootstrap.cc:492] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: No bootstrap required, opened a new log
I20260806 09:15:53.282290 573 ts_tablet_manager.cc:1397] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260806 09:15:53.282290 570 ts_tablet_manager.cc:1397] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260806 09:15:53.282366 572 tablet_bootstrap.cc:492] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: No bootstrap required, opened a new log
I20260806 09:15:53.282662 572 ts_tablet_manager.cc:1397] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 09:15:53.291337 572 raft_consensus.cc:348] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.291715 572 raft_consensus.cc:374] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:53.291822 572 raft_consensus.cc:729] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0feab77147654933a6bce9853d1e0073, State: Initialized, Role: FOLLOWER
I20260806 09:15:53.292461 572 consensus_queue.cc:260] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.293133 570 raft_consensus.cc:348] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.293623 570 raft_consensus.cc:374] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:53.293795 570 raft_consensus.cc:729] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c298d56975a34558b70663c38dcb8ee1, State: Initialized, Role: FOLLOWER
I20260806 09:15:53.293715 573 raft_consensus.cc:348] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.294207 573 raft_consensus.cc:374] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:53.294237 572 ts_tablet_manager.cc:1428] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260806 09:15:53.294368 573 raft_consensus.cc:729] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e587366475654eb6aaf01aa73a0af39a, State: Initialized, Role: FOLLOWER
I20260806 09:15:53.294562 570 consensus_queue.cc:260] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.294996 573 consensus_queue.cc:260] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.295727 538 heartbeater.cc:503] Master 127.3.69.62:40833 was elected leader, sending a full tablet report...
I20260806 09:15:53.296578 570 ts_tablet_manager.cc:1428] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 09:15:53.298048 308 heartbeater.cc:503] Master 127.3.69.62:40833 was elected leader, sending a full tablet report...
I20260806 09:15:53.299310 573 ts_tablet_manager.cc:1428] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260806 09:15:53.420171 578 raft_consensus.cc:482] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 09:15:53.420203 541 tablet.cc:2369] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:53.420518 578 raft_consensus.cc:504] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.422464 578 leader_election.cc:288] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e587366475654eb6aaf01aa73a0af39a (127.3.69.2:38887), c298d56975a34558b70663c38dcb8ee1 (127.3.69.3:41211)
I20260806 09:15:53.430009 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.423024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:40099 (local address 127.3.69.3:41211)
0806 09:15:53.423242 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.423251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.424162 (+ 911us) server_negotiation.cc:407] Connection header received
0806 09:15:53.424381 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.424385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.424446 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.424546 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:53.425192 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.425900 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.428306 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.429081 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.429228 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.429349 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.429625 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.429756 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:15:53.430032 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.422967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.69.1:39565 (local address 127.3.69.2:38887)
0806 09:15:53.423248 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.423257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.424048 (+ 791us) server_negotiation.cc:407] Connection header received
0806 09:15:53.424350 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.424355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.424433 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.424550 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:15:53.425173 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.425880 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.428338 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.429093 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.429232 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.429341 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.429656 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.429781 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:15:53.430226 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.422913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.2:38887 (local address 127.3.69.1:39565)
0806 09:15:53.423929 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.423945 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:53.424094 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:53.424566 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:53.424570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:53.424606 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:53.425017 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.425031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.426026 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.426032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.428127 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.428142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.429355 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.429366 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.429503 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.429525 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 09:15:53.429628 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 09:15:53.429748 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":140,"spinlock_wait_cycles":3328,"thread_start_us":652,"threads_started":1}
I20260806 09:15:53.430290 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.422982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.3:41211 (local address 127.3.69.1:40099)
0806 09:15:53.424033 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.424053 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:15:53.424229 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:15:53.424570 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:15:53.424575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:15:53.424620 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:15:53.425040 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.425049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.426050 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.426055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.428099 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:15:53.428115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.429297 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.429303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:15:53.429461 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.429485 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:15:53.429585 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 09:15:53.429713 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":59,"spinlock_wait_cycles":2944,"thread_start_us":543,"threads_started":1}
I20260806 09:15:53.431432 461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f356474dcf493284db105095c054bc" candidate_uuid: "0feab77147654933a6bce9853d1e0073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c298d56975a34558b70663c38dcb8ee1" is_pre_election: true
I20260806 09:15:53.431677 32706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f356474dcf493284db105095c054bc" candidate_uuid: "0feab77147654933a6bce9853d1e0073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e587366475654eb6aaf01aa73a0af39a" is_pre_election: true
I20260806 09:15:53.431973 461 raft_consensus.cc:2454] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0feab77147654933a6bce9853d1e0073 in term 0.
I20260806 09:15:53.432219 32706 raft_consensus.cc:2454] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0feab77147654933a6bce9853d1e0073 in term 0.
I20260806 09:15:53.432716 32399 leader_election.cc:302] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0feab77147654933a6bce9853d1e0073, c298d56975a34558b70663c38dcb8ee1; no voters:
I20260806 09:15:53.433101 578 raft_consensus.cc:2781] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:15:53.433262 578 raft_consensus.cc:482] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:15:53.433378 578 raft_consensus.cc:3037] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:53.435644 578 raft_consensus.cc:504] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.436219 578 leader_election.cc:288] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [CANDIDATE]: Term 1 election: Requested vote from peers e587366475654eb6aaf01aa73a0af39a (127.3.69.2:38887), c298d56975a34558b70663c38dcb8ee1 (127.3.69.3:41211)
I20260806 09:15:53.436667 32706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f356474dcf493284db105095c054bc" candidate_uuid: "0feab77147654933a6bce9853d1e0073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e587366475654eb6aaf01aa73a0af39a"
I20260806 09:15:53.436738 461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f356474dcf493284db105095c054bc" candidate_uuid: "0feab77147654933a6bce9853d1e0073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c298d56975a34558b70663c38dcb8ee1"
I20260806 09:15:53.436882 32706 raft_consensus.cc:3037] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:53.436918 461 raft_consensus.cc:3037] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:53.440212 32706 raft_consensus.cc:2454] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0feab77147654933a6bce9853d1e0073 in term 1.
I20260806 09:15:53.440212 461 raft_consensus.cc:2454] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0feab77147654933a6bce9853d1e0073 in term 1.
I20260806 09:15:53.440644 32399 leader_election.cc:302] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0feab77147654933a6bce9853d1e0073, c298d56975a34558b70663c38dcb8ee1; no voters:
I20260806 09:15:53.440958 578 raft_consensus.cc:2781] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:53.441414 578 raft_consensus.cc:686] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 1 LEADER]: Becoming Leader. State: Replica: 0feab77147654933a6bce9853d1e0073, State: Running, Role: LEADER
I20260806 09:15:53.441887 578 consensus_queue.cc:237] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } }
I20260806 09:15:53.446846 32276 catalog_manager.cc:5495] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0feab77147654933a6bce9853d1e0073 (127.3.69.1). New cstate: current_term: 1 leader_uuid: "0feab77147654933a6bce9853d1e0073" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0feab77147654933a6bce9853d1e0073" member_type: VOTER last_known_addr { host: "127.3.69.1" port: 46197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 } health_report { overall_health: UNKNOWN } } }
W20260806 09:15:53.450776 309 tablet.cc:2369] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:15:53.455456 32583 tablet.cc:2369] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:15:53.486023 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.480370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.3.69.1:46197)
0806 09:15:53.480586 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.480602 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.480626 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:15:53.480738 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.480742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.480833 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.480997 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:15:53.481391 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.482121 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.483819 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.484287 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.484431 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.484932 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:15:53.485455 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:15:53.485637 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.485690 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.485770 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:15:53.494836 32706 raft_consensus.cc:1261] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 1 FOLLOWER]: Refusing update from remote peer 0feab77147654933a6bce9853d1e0073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:53.494879 461 raft_consensus.cc:1261] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 1 FOLLOWER]: Refusing update from remote peer 0feab77147654933a6bce9853d1e0073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:15:53.495880 585 consensus_queue.cc:1035] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c298d56975a34558b70663c38dcb8ee1" member_type: VOTER last_known_addr { host: "127.3.69.3" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:53.496285 578 consensus_queue.cc:1035] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e587366475654eb6aaf01aa73a0af39a" member_type: VOTER last_known_addr { host: "127.3.69.2" port: 38887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:15:53.508067 590 mvcc.cc:205] Tried to move back new op lower bound from 7315487758304796672 to 7315487758101827584. Current Snapshot: MvccSnapshot[applied={T|T < 7315487758304796672}]
I20260806 09:15:53.510313 593 mvcc.cc:205] Tried to move back new op lower bound from 7315487758304796672 to 7315487758101827584. Current Snapshot: MvccSnapshot[applied={T|T < 7315487758304796672}]
I20260806 09:15:53.523929 32276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:53.524228 32276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:15:53.527689 32276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a75b9fee4e184feda020749c25dc9a14: Sending DeleteTablet for 3 replicas of tablet c2f356474dcf493284db105095c054bc
I20260806 09:15:53.528784 32674 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f356474dcf493284db105095c054bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:53 UTC) from {username='slave'} at 127.0.0.1:55272
I20260806 09:15:53.528837 32473 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f356474dcf493284db105095c054bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:53 UTC) from {username='slave'} at 127.0.0.1:52906
I20260806 09:15:53.529579 425 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f356474dcf493284db105095c054bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:15:53 UTC) from {username='slave'} at 127.0.0.1:39400
I20260806 09:15:53.529731 610 tablet_replica.cc:330] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: stopping tablet replica
I20260806 09:15:53.530139 610 raft_consensus.cc:2229] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:53.530736 610 raft_consensus.cc:2258] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:53.532418 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 32377
I20260806 09:15:53.533442 610 ts_tablet_manager.cc:1905] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:53.541630 612 tablet_replica.cc:330] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: stopping tablet replica
I20260806 09:15:53.541733 611 tablet_replica.cc:330] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: stopping tablet replica
I20260806 09:15:53.542282 611 raft_consensus.cc:2229] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:15:53.542788 612 raft_consensus.cc:2229] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:15:53.543174 611 raft_consensus.cc:2258] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:53.544355 610 ts_tablet_manager.cc:1918] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:53.544600 610 log.cc:1196] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-1/wal/wals/c2f356474dcf493284db105095c054bc
I20260806 09:15:53.545301 610 ts_tablet_manager.cc:1939] T c2f356474dcf493284db105095c054bc P e587366475654eb6aaf01aa73a0af39a: Deleting consensus metadata
I20260806 09:15:53.545593 611 ts_tablet_manager.cc:1905] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:53.546183 612 raft_consensus.cc:2258] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:15:53.547356 32255 catalog_manager.cc:4841] TS e587366475654eb6aaf01aa73a0af39a (127.3.69.2:38887): tablet c2f356474dcf493284db105095c054bc (table test-table [id=457dc428bcbe4a23b631b4f443ce1cc5]) successfully deleted
I20260806 09:15:53.549703 612 ts_tablet_manager.cc:1905] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:15:53.552882 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.535386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.3.69.1:46197)
0806 09:15:53.535559 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.535565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.544035 (+ 8470us) server_negotiation.cc:407] Connection header received
0806 09:15:53.544373 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.544381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.544487 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.544573 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:53.545350 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.546474 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.548645 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.549358 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.549463 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.551027 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:53.551037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:53.551054 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:53.551929 (+ 875us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:53.551988 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:53.552016 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:53.552096 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.552354 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.552474 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 09:15:53.554176 32572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:53.557049 612 ts_tablet_manager.cc:1918] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:53.557278 612 log.cc:1196] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-2/wal/wals/c2f356474dcf493284db105095c054bc
I20260806 09:15:53.557891 612 ts_tablet_manager.cc:1939] T c2f356474dcf493284db105095c054bc P c298d56975a34558b70663c38dcb8ee1: Deleting consensus metadata
I20260806 09:15:53.559836 32251 catalog_manager.cc:4841] TS c298d56975a34558b70663c38dcb8ee1 (127.3.69.3:41211): tablet c2f356474dcf493284db105095c054bc (table test-table [id=457dc428bcbe4a23b631b4f443ce1cc5]) successfully deleted
I20260806 09:15:53.660490 611 ts_tablet_manager.cc:1918] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:15:53.660878 611 log.cc:1196] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Deleting WAL directory at /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786007418649204-3348-0/minicluster-data/ts-0/wal/wals/c2f356474dcf493284db105095c054bc
I20260806 09:15:53.661265 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 32377
I20260806 09:15:53.661639 611 ts_tablet_manager.cc:1939] T c2f356474dcf493284db105095c054bc P 0feab77147654933a6bce9853d1e0073: Deleting consensus metadata
W20260806 09:15:53.692327 32253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.69.1:46197 (error 108)
I20260806 09:15:53.692803 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 32589
I20260806 09:15:53.694293 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.693758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46197 (local address 127.0.0.1:52946)
0806 09:15:53.694024 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.694137 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260806 09:15:53.694697 32253 catalog_manager.cc:4584] TS 0feab77147654933a6bce9853d1e0073 (127.3.69.1:46197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2f356474dcf493284db105095c054bc: Network error: Client connection negotiation failed: client connection to 127.3.69.1:46197: connect: Connection refused (error 111)
I20260806 09:15:53.700281 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.693638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.3.69.2:38887)
0806 09:15:53.693848 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.693853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.693877 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:15:53.694138 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.694143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.694231 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.694340 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:53.694789 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.695649 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.697047 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.697490 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.697631 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.698786 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:53.698795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:53.698807 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:53.699472 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:53.699529 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:53.699550 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:53.699687 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.699821 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.699936 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:15:53.701066 301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:53.748589 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.748150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46197 (local address 127.0.0.1:52950)
0806 09:15:53.748346 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.748440 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 09:15:53.749877 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.749366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46197 (local address 127.0.0.1:52952)
0806 09:15:53.749568 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.749679 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 09:15:53.792227 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 32589
I20260806 09:15:53.810732 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.810219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46197 (local address 127.0.0.1:52960)
0806 09:15:53.810435 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.810546 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 09:15:53.811937 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.811575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46197 (local address 127.0.0.1:52962)
0806 09:15:53.811751 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.811821 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 09:15:53.820133 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 313
I20260806 09:15:53.827399 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.821007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.3.69.3:41211)
0806 09:15:53.821196 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:53.821201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:53.821223 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:15:53.821359 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:53.821363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:53.821468 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:53.821563 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:53.821975 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.822839 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.824261 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:53.824697 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:53.824807 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:53.825903 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:53.825912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:53.825928 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:53.826536 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:53.826586 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:53.826612 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:53.826708 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:53.826827 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 09:15:53.826925 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:15:53.828091 528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:53.892486 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.891974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46197 (local address 127.0.0.1:52972)
0806 09:15:53.892196 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.892293 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 09:15:53.893678 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:53.893230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.1:46197 (local address 127.0.0.1:52974)
0806 09:15:53.893454 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 09:15:53.893533 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.1:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 09:15:53.918184 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 313
I20260806 09:15:53.944379 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 32231
I20260806 09:15:53.945578 32321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:54.058842 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 32231
2026-08-06T09:15:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2958 ms)
[----------] 6 tests from EncryptionPolicyTest (23830 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:15:54 dist-test-slave-w4v5 krb5kdc[622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:15:54 dist-test-slave-w4v5 krb5kdc[622](info): set up 2 sockets
Aug 06 09:15:54 dist-test-slave-w4v5 krb5kdc[622](info): commencing operation
krb5kdc: starting...
W20260806 09:15:55.392928 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:15:55 dist-test-slave-w4v5 krb5kdc[622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:15:55.714363 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:56.064846 637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:56.065155 637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:56.065340 637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:56.072742 637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:56.072862 637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:56.072980 637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:56.073068 637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:56.073117 637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:56.083055 637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:56.084434 637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:56.085770 637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:56.098281 647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:56.098228 644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:56.098333 643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:56.104492 637 server_base.cc:1034] running on GCE node
I20260806 09:15:56.105340 637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:56.106925 637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:56.108134 637 hybrid_clock.cc:648] HybridClock initialized: now 1786007756108098 us; error 36 us; skew 500 ppm
I20260806 09:15:56.108521 637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:56 dist-test-slave-w4v5 krb5kdc[622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:56.111876 637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:15:56.113577 637 webserver.cc:460] Webserver started at http://127.0.0.1:39469/ using document root <none> and password file <none>
I20260806 09:15:56.114323 637 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:56.114499 637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:56.114830 637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:56.117864 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "d1bba82542f84de5bdb7c806f849346e"
format_stamp: "Formatted at 2026-08-06 09:15:56 on dist-test-slave-w4v5"
server_key: "36fee6d4b87960d831bc9cf618e127b6"
server_key_iv: "b469d35637ae20aefb1770676ba8d1cb"
server_key_version: "encryptionkey@0"
I20260806 09:15:56.118654 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "d1bba82542f84de5bdb7c806f849346e"
format_stamp: "Formatted at 2026-08-06 09:15:56 on dist-test-slave-w4v5"
server_key: "36fee6d4b87960d831bc9cf618e127b6"
server_key_iv: "b469d35637ae20aefb1770676ba8d1cb"
server_key_version: "encryptionkey@0"
I20260806 09:15:56.123809 637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 09:15:56.127307 661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:56.128582 637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:15:56.128795 637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "d1bba82542f84de5bdb7c806f849346e"
format_stamp: "Formatted at 2026-08-06 09:15:56 on dist-test-slave-w4v5"
server_key: "36fee6d4b87960d831bc9cf618e127b6"
server_key_iv: "b469d35637ae20aefb1770676ba8d1cb"
server_key_version: "encryptionkey@0"
I20260806 09:15:56.128993 637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:56.149328 637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:56.153298 637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:56.153596 637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:56.172906 637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34815
I20260806 09:15:56.172976 750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34815 every 8 connection(s)
I20260806 09:15:56.174290 637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:56.175016 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 637
I20260806 09:15:56.175374 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:15:56.176178 3348 external_mini_cluster.cc:1428] Setting key 1cd4ccfe92534af21b96b6dc32cb0d9c
I20260806 09:15:56.178752 751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:56.189760 751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e: Bootstrap starting.
Aug 06 09:15:56 dist-test-slave-w4v5 krb5kdc[622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:15:56.193328 751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:56.194422 751 log.cc:824] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:56.198133 751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e: No bootstrap required, opened a new log
I20260806 09:15:56.202106 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:56.178115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37313 (local address 127.0.0.1:34815)
0806 09:15:56.178944 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:56.178957 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:56.178997 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:15:56.180728 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:56.180748 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:56.181119 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:56.181520 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:56.183135 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:56.184465 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:56.185791 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:56.186483 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:56.186706 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:56.191373 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:56.191385 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:56.191398 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:56.191433 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:56.194579 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:56.196343 (+ 1764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:56.196353 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:56.196363 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:56.196610 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:56.197365 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:56.197373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:56.197377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:56.198681 (+ 1304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:56.199309 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:56.199902 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 09:15:56.200402 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":347,"threads_started":1}
I20260806 09:15:56.208145 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:15:56.212051 751 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bba82542f84de5bdb7c806f849346e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } }
I20260806 09:15:56.212579 751 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:56.212709 751 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bba82542f84de5bdb7c806f849346e, State: Initialized, Role: FOLLOWER
I20260806 09:15:56.213521 751 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bba82542f84de5bdb7c806f849346e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } }
I20260806 09:15:56.213860 751 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:56.214017 751 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:56.214211 751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:56.218092 751 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bba82542f84de5bdb7c806f849346e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } }
I20260806 09:15:56.218919 751 leader_election.cc:302] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1bba82542f84de5bdb7c806f849346e; no voters:
I20260806 09:15:56.220490 751 leader_election.cc:288] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:15:56.220597 773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:56.222354 773 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [term 1 LEADER]: Becoming Leader. State: Replica: d1bba82542f84de5bdb7c806f849346e, State: Running, Role: LEADER
I20260806 09:15:56.222882 751 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:56.223049 773 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bba82542f84de5bdb7c806f849346e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } }
I20260806 09:15:56.232622 778 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1bba82542f84de5bdb7c806f849346e. Latest consensus state: current_term: 1 leader_uuid: "d1bba82542f84de5bdb7c806f849346e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bba82542f84de5bdb7c806f849346e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } } }
I20260806 09:15:56.232697 774 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1bba82542f84de5bdb7c806f849346e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1bba82542f84de5bdb7c806f849346e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34815 } } }
I20260806 09:15:56.233294 774 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:56.233446 778 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:56.234270 788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:56.240959 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:56.215020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48699 (local address 127.0.0.1:34815)
0806 09:15:56.215266 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:56.215279 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:56.215756 (+ 477us) server_negotiation.cc:407] Connection header received
0806 09:15:56.215997 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:56.216003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:56.216108 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:56.216238 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:56.218459 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:56.219382 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:56.221129 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:56.221940 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:56.222091 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:56.226359 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:56.226369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:56.226377 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:15:56.226432 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:56.231717 (+ 5285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:56.232977 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:56.232982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:56.232985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:56.233327 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:15:56.233990 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:15:56.233995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:15:56.233998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:15:56.234436 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:56.234733 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:56.236120 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0806 09:15:56.236329 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 09:15:56.244796 788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:56.255369 788 catalog_manager.cc:1347] Generated new cluster ID: 7598eb6a79d74713aa3fc5463438adf7
I20260806 09:15:56.255486 788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:56.278664 788 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:15:56.280114 788 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:56.295679 788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1bba82542f84de5bdb7c806f849346e: Generated new TSK 0
I20260806 09:15:56.296751 788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:15:56.304507 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 637
I20260806 09:15:56.305891 742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:15:56.397758 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 637
W20260806 09:15:56.422753 620 connection.cc:535] client connection to 127.0.0.1:34815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34815): Connection reset by peer (error 104)
2026-08-06T09:15:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:15:56 dist-test-slave-w4v5 krb5kdc[804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:15:56 dist-test-slave-w4v5 krb5kdc[804](info): set up 2 sockets
Aug 06 09:15:56 dist-test-slave-w4v5 krb5kdc[804](info): commencing operation
krb5kdc: starting...
W20260806 09:15:57.718745 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:15:57 dist-test-slave-w4v5 krb5kdc[804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:15:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:15:58.054687 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:15:58.409015 828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:15:58.409348 828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:15:58.409505 828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:15:58.416857 828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:15:58.417025 828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:15:58.417127 828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:15:58.417236 828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:15:58.417312 828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:15:58.427414 828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:15:58.428758 828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:15:58.430109 828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:15:58.442147 836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:58.442121 837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:15:58.442368 840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:15:58.443452 828 server_base.cc:1034] running on GCE node
I20260806 09:15:58.444255 828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:15:58.445575 828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:15:58.446805 828 hybrid_clock.cc:648] HybridClock initialized: now 1786007758446758 us; error 43 us; skew 500 ppm
I20260806 09:15:58.447263 828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:15:58 dist-test-slave-w4v5 krb5kdc[804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:15:58.454011 828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:15:58.455713 828 webserver.cc:460] Webserver started at http://127.0.0.1:44505/ using document root <none> and password file <none>
I20260806 09:15:58.456416 828 fs_manager.cc:359] Metadata directory not provided
I20260806 09:15:58.456570 828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:15:58.456876 828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:15:58.460003 828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "4aba64730ddb4b969d5fccd866d048d4"
format_stamp: "Formatted at 2026-08-06 09:15:58 on dist-test-slave-w4v5"
server_key: "c3fbe7f19aaef6598105d38e82a23078"
server_key_iv: "87726824ac9df688883a59f4dcefccf1"
server_key_version: "encryptionkey@0"
I20260806 09:15:58.460794 828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "4aba64730ddb4b969d5fccd866d048d4"
format_stamp: "Formatted at 2026-08-06 09:15:58 on dist-test-slave-w4v5"
server_key: "c3fbe7f19aaef6598105d38e82a23078"
server_key_iv: "87726824ac9df688883a59f4dcefccf1"
server_key_version: "encryptionkey@0"
I20260806 09:15:58.466048 828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:15:58.469628 852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:15:58.471032 828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:15:58.471271 828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "4aba64730ddb4b969d5fccd866d048d4"
format_stamp: "Formatted at 2026-08-06 09:15:58 on dist-test-slave-w4v5"
server_key: "c3fbe7f19aaef6598105d38e82a23078"
server_key_iv: "87726824ac9df688883a59f4dcefccf1"
server_key_version: "encryptionkey@0"
I20260806 09:15:58.471477 828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:15:58.493503 828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:15:58.497592 828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:15:58.497881 828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:15:58.517189 828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32915
I20260806 09:15:58.517221 933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32915 every 8 connection(s)
I20260806 09:15:58.518611 828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:15:58.519609 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 828
I20260806 09:15:58.519814 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:15:58.520256 3348 external_mini_cluster.cc:1428] Setting key e9d1cddbb084dc73ab2ff9a4a8881a52
I20260806 09:15:58.522939 935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:15:58.534592 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:58.522100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49565 (local address 127.0.0.1:32915)
0806 09:15:58.522906 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:58.522917 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:58.522961 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:15:58.524736 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:58.524756 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:58.525116 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:58.525450 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:58.526098 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:58.527815 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:58.529902 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:58.530591 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:58.530761 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:58.532237 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:58.532244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:58.532257 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:58.532830 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:58.532908 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:58.532936 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:58.533055 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:58.533391 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 09:15:58.533597 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":387,"threads_started":1}
I20260806 09:15:58.535983 935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4: Bootstrap starting.
I20260806 09:15:58.538496 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:15:58.539635 935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4: Neither blocks nor log segments found. Creating new log.
I20260806 09:15:58.540655 935 log.cc:824] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:15:58.543799 935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4: No bootstrap required, opened a new log
I20260806 09:15:58.550654 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:15:58.543283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50807 (local address 127.0.0.1:32915)
0806 09:15:58.543497 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:15:58.543505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:15:58.544037 (+ 532us) server_negotiation.cc:407] Connection header received
0806 09:15:58.544230 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:15:58.544237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:15:58.544299 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:15:58.544383 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:15:58.545043 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:58.545856 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:58.547883 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:15:58.548393 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:15:58.548486 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:15:58.549784 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:15:58.549788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:15:58.549792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:15:58.549808 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:15:58.549860 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:15:58.549878 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:15:58.549964 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 09:15:58.550170 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 09:15:58.550217 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:15:58.553432 935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aba64730ddb4b969d5fccd866d048d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32915 } }
I20260806 09:15:58.553812 935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:15:58.553936 935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aba64730ddb4b969d5fccd866d048d4, State: Initialized, Role: FOLLOWER
I20260806 09:15:58.554514 935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aba64730ddb4b969d5fccd866d048d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32915 } }
I20260806 09:15:58.554744 935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:15:58.554904 935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:15:58.555158 935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:15:58.557366 935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aba64730ddb4b969d5fccd866d048d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32915 } }
I20260806 09:15:58.557871 935 leader_election.cc:302] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aba64730ddb4b969d5fccd866d048d4; no voters:
I20260806 09:15:58.558460 935 leader_election.cc:288] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:15:58.558598 951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:15:58.559329 951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [term 1 LEADER]: Becoming Leader. State: Replica: 4aba64730ddb4b969d5fccd866d048d4, State: Running, Role: LEADER
I20260806 09:15:58.559885 951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aba64730ddb4b969d5fccd866d048d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32915 } }
I20260806 09:15:58.560726 935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:15:58.562902 955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aba64730ddb4b969d5fccd866d048d4. Latest consensus state: current_term: 1 leader_uuid: "4aba64730ddb4b969d5fccd866d048d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aba64730ddb4b969d5fccd866d048d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32915 } } }
I20260806 09:15:58.563416 955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:58.563717 952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aba64730ddb4b969d5fccd866d048d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aba64730ddb4b969d5fccd866d048d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32915 } } }
I20260806 09:15:58.564163 952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:15:58.566752 962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:15:58.571772 962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:15:58.580041 962 catalog_manager.cc:1347] Generated new cluster ID: 1075e6e91ed243f69d2085035b8aa5dc
I20260806 09:15:58.580157 962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:15:58.594674 962 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:15:58.596051 962 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:15:58.606350 962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aba64730ddb4b969d5fccd866d048d4: Generated new TSK 0
I20260806 09:15:58.607236 962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:15:58.622262 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 828
W20260806 09:15:58.623571 925 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49565
W20260806 09:15:58.624377 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:15:58.624646 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 828
W20260806 09:15:58.651580 801 connection.cc:535] client connection to 127.0.0.1:32915 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32915): Connection reset by peer (error 104)
2026-08-06T09:15:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:15:58 dist-test-slave-w4v5 krb5kdc[986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:15:58 dist-test-slave-w4v5 krb5kdc[986](info): set up 2 sockets
Aug 06 09:15:58 dist-test-slave-w4v5 krb5kdc[986](info): commencing operation
krb5kdc: starting...
W20260806 09:15:59.943195 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:00 dist-test-slave-w4v5 krb5kdc[986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:00.278085 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:00.624665 1004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:00.625013 1004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:00.625290 1004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:00.634060 1004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:00.634222 1004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:00.634328 1004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:00.634440 1004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:00.634531 1004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:00.644657 1004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:00.646023 1004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:00.647480 1004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:00.660382 1016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:00.660420 1015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:00.660331 1020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:00.661186 1004 server_base.cc:1034] running on GCE node
I20260806 09:16:00.662115 1004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:00.663661 1004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:00.664916 1004 hybrid_clock.cc:648] HybridClock initialized: now 1786007760664877 us; error 48 us; skew 500 ppm
I20260806 09:16:00.665288 1004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:00 dist-test-slave-w4v5 krb5kdc[986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:00.668723 1004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:00.670543 1004 webserver.cc:460] Webserver started at http://127.0.0.1:37365/ using document root <none> and password file <none>
I20260806 09:16:00.671329 1004 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:00.671487 1004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:00.671806 1004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:00.674836 1004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "dc04397942b546aab33e21b821c71c46"
format_stamp: "Formatted at 2026-08-06 09:16:00 on dist-test-slave-w4v5"
server_key: "d092e64d05f77b2a9a66ad32a1417ab0"
server_key_iv: "6d6e07c02256c6b44b9ee5ea08f9c469"
server_key_version: "encryptionkey@0"
I20260806 09:16:00.675688 1004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "dc04397942b546aab33e21b821c71c46"
format_stamp: "Formatted at 2026-08-06 09:16:00 on dist-test-slave-w4v5"
server_key: "d092e64d05f77b2a9a66ad32a1417ab0"
server_key_iv: "6d6e07c02256c6b44b9ee5ea08f9c469"
server_key_version: "encryptionkey@0"
I20260806 09:16:00.680962 1004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:16:00.684527 1030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:00.685781 1004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:16:00.685966 1004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "dc04397942b546aab33e21b821c71c46"
format_stamp: "Formatted at 2026-08-06 09:16:00 on dist-test-slave-w4v5"
server_key: "d092e64d05f77b2a9a66ad32a1417ab0"
server_key_iv: "6d6e07c02256c6b44b9ee5ea08f9c469"
server_key_version: "encryptionkey@0"
I20260806 09:16:00.686168 1004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:00.692031 1004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:00.696082 1004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:00.696388 1004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:00.716040 1004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37199
I20260806 09:16:00.716097 1114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37199 every 8 connection(s)
I20260806 09:16:00.717505 1004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:00.720825 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 1004
I20260806 09:16:00.721028 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:00.721390 1115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:00.721539 3348 external_mini_cluster.cc:1428] Setting key fab8cc672fdd5100b04c87188b6b509a
I20260806 09:16:00.729313 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:00.723352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52371 (local address 127.0.0.1:37199)
0806 09:16:00.724145 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:00.724155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:00.724208 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:16:00.725982 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:00.726003 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:00.726327 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:00.726632 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:00.727796 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:00.727802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:00.727815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:00.727829 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:00.727893 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:00.727915 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:00.728092 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:00.728209 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 09:16:00.728421 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":344,"threads_started":1}
I20260806 09:16:00.732980 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:00.735177 1115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46: Bootstrap starting.
I20260806 09:16:00.739364 1115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:00.740756 1115 log.cc:824] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:00.741011 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:00.738420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55291 (local address 127.0.0.1:37199)
0806 09:16:00.738572 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:00.738576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:00.738924 (+ 348us) server_negotiation.cc:407] Connection header received
0806 09:16:00.739142 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:00.739145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:00.739202 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:00.739286 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:00.740257 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:00.740261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:00.740264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:00.740278 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:00.740317 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:00.740331 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:00.740423 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:00.740577 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 09:16:00.740622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 09:16:00.743767 1115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46: No bootstrap required, opened a new log
I20260806 09:16:00.751173 1115 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc04397942b546aab33e21b821c71c46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37199 } }
I20260806 09:16:00.751569 1115 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:00.751678 1115 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc04397942b546aab33e21b821c71c46, State: Initialized, Role: FOLLOWER
I20260806 09:16:00.752250 1115 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc04397942b546aab33e21b821c71c46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37199 } }
I20260806 09:16:00.752506 1115 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:00.752651 1115 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:00.752818 1115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:00.755154 1115 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc04397942b546aab33e21b821c71c46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37199 } }
I20260806 09:16:00.755690 1115 leader_election.cc:302] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc04397942b546aab33e21b821c71c46; no voters:
I20260806 09:16:00.756294 1115 leader_election.cc:288] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:00.756438 1129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:00.759346 1129 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [term 1 LEADER]: Becoming Leader. State: Replica: dc04397942b546aab33e21b821c71c46, State: Running, Role: LEADER
I20260806 09:16:00.760016 1129 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc04397942b546aab33e21b821c71c46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37199 } }
I20260806 09:16:00.762223 1129 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc04397942b546aab33e21b821c71c46. Latest consensus state: current_term: 1 leader_uuid: "dc04397942b546aab33e21b821c71c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc04397942b546aab33e21b821c71c46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37199 } } }
I20260806 09:16:00.762588 1129 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:00.763266 1131 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc04397942b546aab33e21b821c71c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc04397942b546aab33e21b821c71c46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37199 } } }
I20260806 09:16:00.763628 1131 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:00.764492 1115 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:00.764796 1133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:00.773777 1133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:00.782629 1133 catalog_manager.cc:1347] Generated new cluster ID: 38c85ddddfa642e1affca7d5f6bad454
I20260806 09:16:00.782759 1133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:00.799247 1133 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:00.799762 1133 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:00.809235 1133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc04397942b546aab33e21b821c71c46: Generated new TSK 0
I20260806 09:16:00.810233 1133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:00.819898 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1004
W20260806 09:16:00.821061 1110 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52371
W20260806 09:16:00.821797 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1004: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:00.821969 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 1004
2026-08-06T09:16:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:00 dist-test-slave-w4v5 krb5kdc[1161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:00 dist-test-slave-w4v5 krb5kdc[1161](info): set up 2 sockets
Aug 06 09:16:00 dist-test-slave-w4v5 krb5kdc[1161](info): commencing operation
krb5kdc: starting...
W20260806 09:16:02.166663 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.296s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:02 dist-test-slave-w4v5 krb5kdc[1161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:02.516705 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:02.874605 1187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:02.874894 1187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:02.875123 1187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:02.882324 1187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:02.882427 1187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:02.882525 1187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:02.882639 1187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:02.882705 1187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:02.892637 1187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:02.893942 1187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:02.895371 1187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:02.907220 1201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:02.907287 1196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:02.907248 1194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:02.908777 1187 server_base.cc:1034] running on GCE node
I20260806 09:16:02.909636 1187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:02.911276 1187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:02.912547 1187 hybrid_clock.cc:648] HybridClock initialized: now 1786007762912472 us; error 38 us; skew 500 ppm
I20260806 09:16:02.912891 1187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:02 dist-test-slave-w4v5 krb5kdc[1161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:02.916051 1187 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:02.917611 1187 webserver.cc:460] Webserver started at http://127.0.0.1:41353/ using document root <none> and password file <none>
I20260806 09:16:02.918315 1187 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:02.918452 1187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:02.918785 1187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:02.921904 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "50d0722612f14b949d983248f03378ac"
format_stamp: "Formatted at 2026-08-06 09:16:02 on dist-test-slave-w4v5"
server_key: "43cb4502a052cb09002969d56fe54d61"
server_key_iv: "138cc71704a3f425d763f9301d310a38"
server_key_version: "encryptionkey@0"
I20260806 09:16:02.922700 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "50d0722612f14b949d983248f03378ac"
format_stamp: "Formatted at 2026-08-06 09:16:02 on dist-test-slave-w4v5"
server_key: "43cb4502a052cb09002969d56fe54d61"
server_key_iv: "138cc71704a3f425d763f9301d310a38"
server_key_version: "encryptionkey@0"
I20260806 09:16:02.928000 1187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 09:16:02.931407 1211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:02.932680 1187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 09:16:02.932866 1187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "50d0722612f14b949d983248f03378ac"
format_stamp: "Formatted at 2026-08-06 09:16:02 on dist-test-slave-w4v5"
server_key: "43cb4502a052cb09002969d56fe54d61"
server_key_iv: "138cc71704a3f425d763f9301d310a38"
server_key_version: "encryptionkey@0"
I20260806 09:16:02.933064 1187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:02.949990 1187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:02.953987 1187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:02.954280 1187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:02.973276 1187 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35519
I20260806 09:16:02.973320 1281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35519 every 8 connection(s)
I20260806 09:16:02.974705 1187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:02.978451 1283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:02.980450 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 1187
I20260806 09:16:02.980640 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:02.981226 3348 external_mini_cluster.cc:1428] Setting key 69e16f288a78e1232a0343ff45cf674b
I20260806 09:16:02.992182 1283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac: Bootstrap starting.
I20260806 09:16:02.995590 1283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac: Neither blocks nor log segments found. Creating new log.
Aug 06 09:16:02 dist-test-slave-w4v5 krb5kdc[1161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:16:02.997006 1283 log.cc:824] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:03.000360 1283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac: No bootstrap required, opened a new log
I20260806 09:16:03.005266 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:02.983217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41207 (local address 127.0.0.1:35519)
0806 09:16:02.984040 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:02.984063 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:02.984105 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:16:02.985741 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:02.985763 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:02.986154 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:02.986538 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:02.988191 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:02.989560 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:02.991471 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:02.992150 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:02.992355 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:02.997348 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:02.997356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:02.997370 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:02.997409 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:03.000883 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:03.001806 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:03.001813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:03.001819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:03.001986 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:03.002411 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:03.002414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:03.002416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:03.003001 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:03.003346 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:03.003772 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 09:16:03.004180 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":363,"threads_started":1}
I20260806 09:16:03.009558 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:03.010113 1283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d0722612f14b949d983248f03378ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } }
I20260806 09:16:03.010576 1283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:03.010685 1283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d0722612f14b949d983248f03378ac, State: Initialized, Role: FOLLOWER
I20260806 09:16:03.011371 1283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d0722612f14b949d983248f03378ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } }
I20260806 09:16:03.011667 1283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:03.011798 1283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:03.011968 1283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:03.015477 1283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d0722612f14b949d983248f03378ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } }
I20260806 09:16:03.016165 1283 leader_election.cc:302] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50d0722612f14b949d983248f03378ac; no voters:
I20260806 09:16:03.016825 1283 leader_election.cc:288] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:03.016989 1298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:03.019174 1283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:03.023645 1298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [term 1 LEADER]: Becoming Leader. State: Replica: 50d0722612f14b949d983248f03378ac, State: Running, Role: LEADER
I20260806 09:16:03.024379 1298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d0722612f14b949d983248f03378ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } }
I20260806 09:16:03.027628 1300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50d0722612f14b949d983248f03378ac. Latest consensus state: current_term: 1 leader_uuid: "50d0722612f14b949d983248f03378ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d0722612f14b949d983248f03378ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } } }
I20260806 09:16:03.028097 1300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:03.029700 1307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:03.029526 1299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50d0722612f14b949d983248f03378ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d0722612f14b949d983248f03378ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35519 } } }
I20260806 09:16:03.030623 1299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:03.036229 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:03.015103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34821 (local address 127.0.0.1:35519)
0806 09:16:03.015313 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:03.015321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:03.015899 (+ 578us) server_negotiation.cc:407] Connection header received
0806 09:16:03.016040 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:03.016046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:03.016127 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:03.016256 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:03.018414 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:03.019548 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:03.021966 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:03.022924 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:03.023130 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:03.026739 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:03.026751 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:03.026756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:03.026804 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:03.031443 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:03.032389 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:03.032398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:03.032402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:03.032571 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:03.033106 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:03.033114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:03.033120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:03.033794 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:03.034142 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:03.035374 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0806 09:16:03.035687 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 09:16:03.040269 1307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:03.048974 1307 catalog_manager.cc:1347] Generated new cluster ID: ac3bf2f19238449a8a90942d9b22a04c
I20260806 09:16:03.049117 1307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:03.104528 1307 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:03.105926 1307 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:03.120126 1307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50d0722612f14b949d983248f03378ac: Generated new TSK 0
I20260806 09:16:03.121093 1307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:03.131346 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1187
I20260806 09:16:03.132457 1274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:03.232980 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 1187
2026-08-06T09:16:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:03 dist-test-slave-w4v5 krb5kdc[1335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:03 dist-test-slave-w4v5 krb5kdc[1335](info): set up 2 sockets
Aug 06 09:16:03 dist-test-slave-w4v5 krb5kdc[1335](info): commencing operation
krb5kdc: starting...
W20260806 09:16:04.553979 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:04 dist-test-slave-w4v5 krb5kdc[1335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:04.902962 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:05.254593 1355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:05.254884 1355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:05.255059 1355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:05.262274 1355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:05.262377 1355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:05.262475 1355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:05.262581 1355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:05.262647 1355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:05.272550 1355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:05.273911 1355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:05.275292 1355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:05.287149 1366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:05.287304 1364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:05.287796 1363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:05.288635 1355 server_base.cc:1034] running on GCE node
I20260806 09:16:05.289482 1355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:05.290825 1355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:05.292085 1355 hybrid_clock.cc:648] HybridClock initialized: now 1786007765292013 us; error 61 us; skew 500 ppm
I20260806 09:16:05.292500 1355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:05 dist-test-slave-w4v5 krb5kdc[1335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:05.298965 1355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:05.300521 1355 webserver.cc:460] Webserver started at http://127.0.0.1:45323/ using document root <none> and password file <none>
I20260806 09:16:05.301205 1355 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:05.301309 1355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:05.301563 1355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:05.304658 1355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "2251cfeb0c8e4710a132a7c6a3ad4444"
format_stamp: "Formatted at 2026-08-06 09:16:05 on dist-test-slave-w4v5"
server_key: "ccb73612afefeb1bd6eaa0ac8a490048"
server_key_iv: "a7c2b0756111bc42e1904eae7e4319c2"
server_key_version: "encryptionkey@0"
I20260806 09:16:05.305365 1355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "2251cfeb0c8e4710a132a7c6a3ad4444"
format_stamp: "Formatted at 2026-08-06 09:16:05 on dist-test-slave-w4v5"
server_key: "ccb73612afefeb1bd6eaa0ac8a490048"
server_key_iv: "a7c2b0756111bc42e1904eae7e4319c2"
server_key_version: "encryptionkey@0"
I20260806 09:16:05.310585 1355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 09:16:05.314209 1375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:05.315681 1355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 09:16:05.315878 1355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "2251cfeb0c8e4710a132a7c6a3ad4444"
format_stamp: "Formatted at 2026-08-06 09:16:05 on dist-test-slave-w4v5"
server_key: "ccb73612afefeb1bd6eaa0ac8a490048"
server_key_iv: "a7c2b0756111bc42e1904eae7e4319c2"
server_key_version: "encryptionkey@0"
I20260806 09:16:05.316073 1355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:05.337049 1355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:05.341305 1355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:05.341601 1355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:05.361275 1355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42513
I20260806 09:16:05.361330 1465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42513 every 8 connection(s)
I20260806 09:16:05.362732 1355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:05.366544 1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:05.368572 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 1355
I20260806 09:16:05.368784 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:05.369377 3348 external_mini_cluster.cc:1428] Setting key e69d1c3885c5c131fcc08a86a0632a62
I20260806 09:16:05.379783 1467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444: Bootstrap starting.
I20260806 09:16:05.382233 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:05.371354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38483 (local address 127.0.0.1:42513)
0806 09:16:05.372169 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:05.372181 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:05.372227 (+ 46us) server_negotiation.cc:407] Connection header received
0806 09:16:05.373820 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:05.373841 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:05.374224 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:05.374623 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:05.375167 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:05.376529 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:05.377935 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:05.378575 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:05.378789 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:05.380050 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:05.380057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:05.380070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:05.380579 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:05.380663 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:05.380690 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:05.380821 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:05.381072 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 09:16:05.381327 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":362,"threads_started":1}
I20260806 09:16:05.383013 1467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:05.384131 1467 log.cc:824] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:05.385885 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:05.387042 1467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444: No bootstrap required, opened a new log
I20260806 09:16:05.396816 1467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2251cfeb0c8e4710a132a7c6a3ad4444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42513 } }
I20260806 09:16:05.397394 1467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:05.397487 1467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2251cfeb0c8e4710a132a7c6a3ad4444, State: Initialized, Role: FOLLOWER
I20260806 09:16:05.397540 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:05.390309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49075 (local address 127.0.0.1:42513)
0806 09:16:05.390530 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:05.390536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:05.391155 (+ 619us) server_negotiation.cc:407] Connection header received
0806 09:16:05.391357 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:05.391362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:05.391459 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:05.391566 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:05.392223 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:05.392985 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:05.394924 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:05.395456 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:05.395554 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:05.396723 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:05.396727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:05.396730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:05.396746 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:05.396795 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:05.396813 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:05.396881 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:05.397054 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 09:16:05.397126 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 09:16:05.398244 1467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2251cfeb0c8e4710a132a7c6a3ad4444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42513 } }
I20260806 09:16:05.398608 1467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:05.398763 1467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:05.398957 1467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:05.401516 1467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2251cfeb0c8e4710a132a7c6a3ad4444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42513 } }
I20260806 09:16:05.402015 1467 leader_election.cc:302] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2251cfeb0c8e4710a132a7c6a3ad4444; no voters:
I20260806 09:16:05.402609 1467 leader_election.cc:288] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:05.402788 1483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:05.403551 1483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [term 1 LEADER]: Becoming Leader. State: Replica: 2251cfeb0c8e4710a132a7c6a3ad4444, State: Running, Role: LEADER
I20260806 09:16:05.404158 1483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2251cfeb0c8e4710a132a7c6a3ad4444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42513 } }
I20260806 09:16:05.404795 1467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:05.407402 1486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2251cfeb0c8e4710a132a7c6a3ad4444. Latest consensus state: current_term: 1 leader_uuid: "2251cfeb0c8e4710a132a7c6a3ad4444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2251cfeb0c8e4710a132a7c6a3ad4444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42513 } } }
I20260806 09:16:05.407528 1484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2251cfeb0c8e4710a132a7c6a3ad4444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2251cfeb0c8e4710a132a7c6a3ad4444" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42513 } } }
I20260806 09:16:05.407886 1486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:05.407958 1484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:05.409901 1498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:05.415223 1498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:05.424325 1498 catalog_manager.cc:1347] Generated new cluster ID: fbf25b11dd47480cb656082e1ad3a59b
I20260806 09:16:05.424491 1498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:05.441947 1498 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:05.443377 1498 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:05.453737 1498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2251cfeb0c8e4710a132a7c6a3ad4444: Generated new TSK 0
I20260806 09:16:05.454571 1498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:05.465023 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1355
W20260806 09:16:05.466166 1455 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38483
W20260806 09:16:05.466881 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:05.467037 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 1355
2026-08-06T09:16:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:05 dist-test-slave-w4v5 krb5kdc[1517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:05 dist-test-slave-w4v5 krb5kdc[1517](info): set up 2 sockets
Aug 06 09:16:05 dist-test-slave-w4v5 krb5kdc[1517](info): commencing operation
krb5kdc: starting...
W20260806 09:16:06.798472 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.284s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:07 dist-test-slave-w4v5 krb5kdc[1517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:07.154664 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:07.501746 1540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:07.502058 1540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:07.502156 1540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:07.509466 1540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:07.509596 1540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:07.509647 1540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:07.509711 1540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:07.509753 1540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:07.519761 1540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:07.521075 1540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:07.522387 1540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:07.534683 1553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:07.534788 1548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:07.534782 1550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:07.535949 1540 server_base.cc:1034] running on GCE node
I20260806 09:16:07.536729 1540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:07.538158 1540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:07.539392 1540 hybrid_clock.cc:648] HybridClock initialized: now 1786007767539345 us; error 59 us; skew 500 ppm
I20260806 09:16:07.539748 1540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:07 dist-test-slave-w4v5 krb5kdc[1517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:07.542917 1540 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:07.544395 1540 webserver.cc:460] Webserver started at http://127.0.0.1:37175/ using document root <none> and password file <none>
I20260806 09:16:07.545085 1540 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:07.545238 1540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:07.545562 1540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:07.548688 1540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "45f6dae529b54b1593e8887f6f332418"
format_stamp: "Formatted at 2026-08-06 09:16:07 on dist-test-slave-w4v5"
server_key: "95416309e2b15301471ba9eab35a5363"
server_key_iv: "9257d3bfeb8968eec3e4ca27d3d6d3a9"
server_key_version: "encryptionkey@0"
I20260806 09:16:07.549470 1540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "45f6dae529b54b1593e8887f6f332418"
format_stamp: "Formatted at 2026-08-06 09:16:07 on dist-test-slave-w4v5"
server_key: "95416309e2b15301471ba9eab35a5363"
server_key_iv: "9257d3bfeb8968eec3e4ca27d3d6d3a9"
server_key_version: "encryptionkey@0"
I20260806 09:16:07.554636 1540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 09:16:07.558211 1568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:07.559579 1540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:16:07.559787 1540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "45f6dae529b54b1593e8887f6f332418"
format_stamp: "Formatted at 2026-08-06 09:16:07 on dist-test-slave-w4v5"
server_key: "95416309e2b15301471ba9eab35a5363"
server_key_iv: "9257d3bfeb8968eec3e4ca27d3d6d3a9"
server_key_version: "encryptionkey@0"
I20260806 09:16:07.559993 1540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:07.565654 1540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:07.570243 1540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:07.570562 1540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:07.590283 1540 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40473
I20260806 09:16:07.590327 1652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40473 every 8 connection(s)
I20260806 09:16:07.591815 1540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:07.595641 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:07.598680 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 1540
I20260806 09:16:07.598902 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:07.599457 3348 external_mini_cluster.cc:1428] Setting key bf6b4923c89b792b6d3183c099707949
I20260806 09:16:07.607193 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:07.601208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46065 (local address 127.0.0.1:40473)
0806 09:16:07.602010 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:07.602022 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:07.602064 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:16:07.603755 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:07.603777 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:07.604122 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:07.604442 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:07.605584 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:07.605590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:07.605633 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:07.605651 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:07.605707 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:07.605731 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:07.605909 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:07.606023 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 09:16:07.606214 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":383,"threads_started":1}
I20260806 09:16:07.607887 1653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418: Bootstrap starting.
I20260806 09:16:07.611167 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:07.611678 1653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:07.612917 1653 log.cc:824] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:07.616066 1653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418: No bootstrap required, opened a new log
I20260806 09:16:07.621743 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:07.616544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40339 (local address 127.0.0.1:40473)
0806 09:16:07.616724 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:07.616728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:07.619674 (+ 2946us) server_negotiation.cc:407] Connection header received
0806 09:16:07.619900 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:07.619912 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:07.619960 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:07.620102 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:07.621046 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:07.621051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:07.621056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:07.621075 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:07.621124 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:07.621139 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:07.621266 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:07.621391 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 09:16:07.621458 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 09:16:07.625842 1653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f6dae529b54b1593e8887f6f332418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } }
I20260806 09:16:07.626250 1653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:07.626371 1653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f6dae529b54b1593e8887f6f332418, State: Initialized, Role: FOLLOWER
I20260806 09:16:07.626976 1653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f6dae529b54b1593e8887f6f332418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } }
I20260806 09:16:07.627298 1653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:07.627465 1653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:07.627647 1653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:07.630255 1653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f6dae529b54b1593e8887f6f332418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } }
I20260806 09:16:07.630896 1653 leader_election.cc:302] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f6dae529b54b1593e8887f6f332418; no voters:
I20260806 09:16:07.631546 1653 leader_election.cc:288] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:07.631699 1667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:07.632438 1667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [term 1 LEADER]: Becoming Leader. State: Replica: 45f6dae529b54b1593e8887f6f332418, State: Running, Role: LEADER
I20260806 09:16:07.633067 1667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f6dae529b54b1593e8887f6f332418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } }
I20260806 09:16:07.633827 1653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:07.636425 1671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f6dae529b54b1593e8887f6f332418. Latest consensus state: current_term: 1 leader_uuid: "45f6dae529b54b1593e8887f6f332418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f6dae529b54b1593e8887f6f332418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } } }
I20260806 09:16:07.636477 1668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f6dae529b54b1593e8887f6f332418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f6dae529b54b1593e8887f6f332418" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } } }
I20260806 09:16:07.636996 1668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:07.637681 1671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:07.637938 1681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:07.647989 1681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:07.658924 1681 catalog_manager.cc:1347] Generated new cluster ID: 5a92bc9022594231a28855f19925f260
I20260806 09:16:07.659047 1681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:07.669317 1681 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:07.669718 1681 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:07.679399 1681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45f6dae529b54b1593e8887f6f332418: Generated new TSK 0
I20260806 09:16:07.680147 1681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:07.688558 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1540
W20260806 09:16:07.689697 1641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46065
W20260806 09:16:07.690407 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1540: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:07.690587 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 1540
2026-08-06T09:16:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:07 dist-test-slave-w4v5 krb5kdc[1708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:07 dist-test-slave-w4v5 krb5kdc[1708](info): set up 2 sockets
Aug 06 09:16:07 dist-test-slave-w4v5 krb5kdc[1708](info): commencing operation
krb5kdc: starting...
W20260806 09:16:09.018155 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.005s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:09 dist-test-slave-w4v5 krb5kdc[1708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:09.354616 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:09.701320 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:09.701609 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:09.701787 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:09.708899 1724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:09.709000 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:09.709126 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:09.709245 1724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:09.709323 1724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:09.719151 1724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:09.720463 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:09.721788 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:09.733783 1733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:09.733933 1739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:09.733902 1737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:09.734882 1724 server_base.cc:1034] running on GCE node
I20260806 09:16:09.735770 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:09.737111 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:09.738372 1724 hybrid_clock.cc:648] HybridClock initialized: now 1786007769738333 us; error 43 us; skew 500 ppm
I20260806 09:16:09.738751 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:09 dist-test-slave-w4v5 krb5kdc[1708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:09.742010 1724 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:09.743584 1724 webserver.cc:460] Webserver started at http://127.0.0.1:36205/ using document root <none> and password file <none>
I20260806 09:16:09.744268 1724 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:09.744417 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:09.744726 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:09.747718 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "ec109a68c52b431f9fe94b1eebc297f5"
format_stamp: "Formatted at 2026-08-06 09:16:09 on dist-test-slave-w4v5"
server_key: "e5ebc98241abeb8243379bf565ceefb6"
server_key_iv: "41364a3a7d8106bdc948a3a6ab602963"
server_key_version: "encryptionkey@0"
I20260806 09:16:09.748487 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "ec109a68c52b431f9fe94b1eebc297f5"
format_stamp: "Formatted at 2026-08-06 09:16:09 on dist-test-slave-w4v5"
server_key: "e5ebc98241abeb8243379bf565ceefb6"
server_key_iv: "41364a3a7d8106bdc948a3a6ab602963"
server_key_version: "encryptionkey@0"
I20260806 09:16:09.753609 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:16:09.757179 1749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:09.758518 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:16:09.758769 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "ec109a68c52b431f9fe94b1eebc297f5"
format_stamp: "Formatted at 2026-08-06 09:16:09 on dist-test-slave-w4v5"
server_key: "e5ebc98241abeb8243379bf565ceefb6"
server_key_iv: "41364a3a7d8106bdc948a3a6ab602963"
server_key_version: "encryptionkey@0"
I20260806 09:16:09.758982 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:09.785045 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:09.789368 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:09.789681 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:09.809324 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36517
I20260806 09:16:09.809358 1834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36517 every 8 connection(s)
I20260806 09:16:09.810765 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:09.814608 1836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:09.816596 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 1724
I20260806 09:16:09.816814 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:09.817287 3348 external_mini_cluster.cc:1428] Setting key cfc1e3a86b81c1a8691db1df4fe4c59c
I20260806 09:16:09.828863 1836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5: Bootstrap starting.
I20260806 09:16:09.832681 1836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:09.833911 1836 log.cc:824] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:16:09 dist-test-slave-w4v5 krb5kdc[1708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:16:09.837684 1836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5: No bootstrap required, opened a new log
I20260806 09:16:09.845355 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:09.819455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:53715 (local address 127.0.0.1:36517)
0806 09:16:09.820309 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:09.820324 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:09.820377 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:16:09.822776 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:09.822808 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:09.823359 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:09.823817 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:09.825283 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:09.827401 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:09.829327 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:09.830322 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:09.830524 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:09.835899 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:09.835909 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:09.835928 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:09.835978 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:09.840328 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:09.841231 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:09.841240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:09.841249 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:09.841464 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:09.841963 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:09.841968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:09.841971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:09.842687 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:09.842994 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:09.843588 (+ 594us) server_negotiation.cc:299] Negotiation successful
0806 09:16:09.843952 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":335,"threads_started":1}
I20260806 09:16:09.848618 1836 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec109a68c52b431f9fe94b1eebc297f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260806 09:16:09.849123 1836 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:09.849256 1836 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec109a68c52b431f9fe94b1eebc297f5, State: Initialized, Role: FOLLOWER
I20260806 09:16:09.849778 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:09.849936 1836 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec109a68c52b431f9fe94b1eebc297f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260806 09:16:09.850203 1836 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:09.850356 1836 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:09.850560 1836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:09.853034 1836 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec109a68c52b431f9fe94b1eebc297f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260806 09:16:09.853693 1836 leader_election.cc:302] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec109a68c52b431f9fe94b1eebc297f5; no voters:
I20260806 09:16:09.854441 1836 leader_election.cc:288] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:09.854524 1847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:09.855329 1847 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [term 1 LEADER]: Becoming Leader. State: Replica: ec109a68c52b431f9fe94b1eebc297f5, State: Running, Role: LEADER
I20260806 09:16:09.855950 1847 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec109a68c52b431f9fe94b1eebc297f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260806 09:16:09.858837 1849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec109a68c52b431f9fe94b1eebc297f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec109a68c52b431f9fe94b1eebc297f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } } }
I20260806 09:16:09.858961 1848 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec109a68c52b431f9fe94b1eebc297f5. Latest consensus state: current_term: 1 leader_uuid: "ec109a68c52b431f9fe94b1eebc297f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec109a68c52b431f9fe94b1eebc297f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } } }
I20260806 09:16:09.859346 1848 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:09.859344 1849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:09.860669 1836 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:09.861405 1855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:09.876343 1855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:09.877108 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:09.854615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:41503 (local address 127.0.0.1:36517)
0806 09:16:09.854824 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:09.854832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:09.858678 (+ 3846us) server_negotiation.cc:407] Connection header received
0806 09:16:09.859867 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:09.859872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:09.859955 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:09.860035 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:09.862946 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:09.864332 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:09.865880 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:09.866300 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:09.866438 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:09.869074 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:09.869079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:09.869082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:09.869113 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:09.872595 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:09.873766 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:09.873773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:09.873781 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:09.873920 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:09.874510 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:09.874516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:09.874519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:09.874988 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:09.876250 (+ 1262us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:09.876507 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 09:16:09.876723 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:16:09.888298 1855 catalog_manager.cc:1347] Generated new cluster ID: 7bab68d698fe43af82c3b2a0039d0b07
I20260806 09:16:09.888418 1855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:09.913759 1855 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:09.915165 1855 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:09.931545 1855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec109a68c52b431f9fe94b1eebc297f5: Generated new TSK 0
I20260806 09:16:09.932358 1855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:09.944051 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1724
I20260806 09:16:09.945240 1827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:10.039394 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 1724
W20260806 09:16:10.066247 1705 connection.cc:535] client connection to 127.0.0.1:36517 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36517): Connection reset by peer (error 104)
2026-08-06T09:16:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:10 dist-test-slave-w4v5 krb5kdc[1884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:10 dist-test-slave-w4v5 krb5kdc[1884](info): set up 2 sockets
Aug 06 09:16:10 dist-test-slave-w4v5 krb5kdc[1884](info): commencing operation
krb5kdc: starting...
W20260806 09:16:11.366942 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:11 dist-test-slave-w4v5 krb5kdc[1884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:11.718600 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:12.069646 1910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:12.069996 1910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:12.070176 1910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:12.077416 1910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:12.077625 1910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:12.077737 1910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:12.077864 1910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:12.077953 1910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:12.087932 1910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:12.089280 1910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:12.090643 1910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:12.103236 1916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:12.103171 1917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:12.103197 1920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:12.103989 1910 server_base.cc:1034] running on GCE node
I20260806 09:16:12.104796 1910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:12.106273 1910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:12.107506 1910 hybrid_clock.cc:648] HybridClock initialized: now 1786007772107462 us; error 61 us; skew 500 ppm
I20260806 09:16:12.107884 1910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:12 dist-test-slave-w4v5 krb5kdc[1884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:12.116854 1910 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:12.118510 1910 webserver.cc:460] Webserver started at http://127.0.0.1:42383/ using document root <none> and password file <none>
I20260806 09:16:12.119290 1910 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:12.119441 1910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:12.119765 1910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:12.122879 1910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1"
format_stamp: "Formatted at 2026-08-06 09:16:12 on dist-test-slave-w4v5"
server_key: "bb8108c80c935596dc57882a3dc54eb6"
server_key_iv: "3fa189947262a3c35273cf8ddd33e344"
server_key_version: "encryptionkey@0"
I20260806 09:16:12.123718 1910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1"
format_stamp: "Formatted at 2026-08-06 09:16:12 on dist-test-slave-w4v5"
server_key: "bb8108c80c935596dc57882a3dc54eb6"
server_key_iv: "3fa189947262a3c35273cf8ddd33e344"
server_key_version: "encryptionkey@0"
I20260806 09:16:12.128875 1910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:16:12.132279 1935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:12.133484 1910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:16:12.133682 1910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1"
format_stamp: "Formatted at 2026-08-06 09:16:12 on dist-test-slave-w4v5"
server_key: "bb8108c80c935596dc57882a3dc54eb6"
server_key_iv: "3fa189947262a3c35273cf8ddd33e344"
server_key_version: "encryptionkey@0"
I20260806 09:16:12.133886 1910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:12.149534 1910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:12.153692 1910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:12.153990 1910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:12.173265 1910 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37269
I20260806 09:16:12.173308 2009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37269 every 8 connection(s)
I20260806 09:16:12.174732 1910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:12.178485 2010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:12.183347 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 1910
I20260806 09:16:12.183537 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:12.184008 3348 external_mini_cluster.cc:1428] Setting key 91ab22e226b97fbcf67da20017ef649c
I20260806 09:16:12.189489 2010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1: Bootstrap starting.
I20260806 09:16:12.193118 2010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:12.194303 2010 log.cc:824] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:12.197408 2010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1: No bootstrap required, opened a new log
I20260806 09:16:12.197969 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:12.185973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:34475 (local address 127.0.0.1:37269)
0806 09:16:12.186862 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:12.186876 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:12.186922 (+ 46us) server_negotiation.cc:407] Connection header received
0806 09:16:12.188694 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:12.188716 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:12.189091 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:12.189398 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:12.190025 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:12.191434 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:12.193450 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:12.194140 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:12.194275 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:12.195733 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:12.195747 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:12.195772 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:12.196289 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:12.196356 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:12.196380 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:12.196507 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:12.196791 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 09:16:12.196997 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":380,"threads_started":1}
I20260806 09:16:12.201792 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:12.205334 2010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260806 09:16:12.205703 2010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:12.205797 2010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0bd6adc0bd4dbe9c14c3321de3f8c1, State: Initialized, Role: FOLLOWER
I20260806 09:16:12.206533 2010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260806 09:16:12.206882 2010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:12.207059 2010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:12.207288 2010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:12.209542 2010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260806 09:16:12.210035 2010 leader_election.cc:302] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d0bd6adc0bd4dbe9c14c3321de3f8c1; no voters:
I20260806 09:16:12.210776 2010 leader_election.cc:288] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:12.210876 2033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:12.213035 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:12.206338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:42255 (local address 127.0.0.1:37269)
0806 09:16:12.206516 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:12.206524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:12.206955 (+ 431us) server_negotiation.cc:407] Connection header received
0806 09:16:12.207200 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:12.207207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:12.207293 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:12.207399 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:12.207926 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:12.208721 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:12.210102 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:12.210684 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:12.210843 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:12.212215 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:12.212219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:12.212223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:12.212239 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:12.212295 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:12.212310 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:12.212444 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:12.212650 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 09:16:12.212698 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 09:16:12.213511 2010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:12.215459 2033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [term 1 LEADER]: Becoming Leader. State: Replica: 9d0bd6adc0bd4dbe9c14c3321de3f8c1, State: Running, Role: LEADER
I20260806 09:16:12.216306 2033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } }
I20260806 09:16:12.219805 2035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d0bd6adc0bd4dbe9c14c3321de3f8c1. Latest consensus state: current_term: 1 leader_uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } } }
I20260806 09:16:12.220227 2035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:12.220978 2034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d0bd6adc0bd4dbe9c14c3321de3f8c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37269 } } }
I20260806 09:16:12.225261 2034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1 [sys.catalog]: This master's current role is: LEADER
W20260806 09:16:12.229848 2056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:16:12.229990 2056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:16:12.230149 2057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:12.231866 2057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:12.240339 2057 catalog_manager.cc:1347] Generated new cluster ID: 85f8820f9861400e834cd76b7f7416c6
I20260806 09:16:12.240520 2057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:12.252591 2057 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:12.253834 2057 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:12.263286 2057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d0bd6adc0bd4dbe9c14c3321de3f8c1: Generated new TSK 0
I20260806 09:16:12.264072 2057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:12.276777 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1910
W20260806 09:16:12.277961 2002 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.75:34475
W20260806 09:16:12.278729 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 1910: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:12.278888 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 1910
W20260806 09:16:12.308002 1880 connection.cc:535] client connection to 127.0.0.1:37269 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37269): Connection reset by peer (error 104)
2026-08-06T09:16:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:12 dist-test-slave-w4v5 krb5kdc[2064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:12 dist-test-slave-w4v5 krb5kdc[2064](info): set up 2 sockets
Aug 06 09:16:12 dist-test-slave-w4v5 krb5kdc[2064](info): commencing operation
krb5kdc: starting...
W20260806 09:16:13.612159 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:13 dist-test-slave-w4v5 krb5kdc[2064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:13.960484 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:14.321100 2084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:14.321424 2084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:14.321599 2084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:14.329231 2084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:14.329356 2084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:14.329470 2084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:14.329584 2084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:14.329666 2084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:14.340356 2084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:14.341758 2084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:14.343119 2084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:14.355696 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:14.355649 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:14.355610 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:14.356616 2084 server_base.cc:1034] running on GCE node
I20260806 09:16:14.357467 2084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:14.358989 2084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:14.360268 2084 hybrid_clock.cc:648] HybridClock initialized: now 1786007774360227 us; error 35 us; skew 500 ppm
I20260806 09:16:14.360670 2084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:14 dist-test-slave-w4v5 krb5kdc[2064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:14.364368 2084 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:14.366164 2084 webserver.cc:460] Webserver started at http://127.0.0.1:36359/ using document root <none> and password file <none>
I20260806 09:16:14.366920 2084 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:14.367108 2084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:14.367466 2084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:14.370689 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "93519348a4ae46efab86c1cb4ac5a628"
format_stamp: "Formatted at 2026-08-06 09:16:14 on dist-test-slave-w4v5"
server_key: "ea2794fdcceeef623b2ecb6663f4f09a"
server_key_iv: "88f925b8aa093f7b6d9ffbc41c40b3f3"
server_key_version: "encryptionkey@0"
I20260806 09:16:14.371579 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "93519348a4ae46efab86c1cb4ac5a628"
format_stamp: "Formatted at 2026-08-06 09:16:14 on dist-test-slave-w4v5"
server_key: "ea2794fdcceeef623b2ecb6663f4f09a"
server_key_iv: "88f925b8aa093f7b6d9ffbc41c40b3f3"
server_key_version: "encryptionkey@0"
I20260806 09:16:14.377094 2084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:16:14.380676 2111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:14.381945 2084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:16:14.382135 2084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "93519348a4ae46efab86c1cb4ac5a628"
format_stamp: "Formatted at 2026-08-06 09:16:14 on dist-test-slave-w4v5"
server_key: "ea2794fdcceeef623b2ecb6663f4f09a"
server_key_iv: "88f925b8aa093f7b6d9ffbc41c40b3f3"
server_key_version: "encryptionkey@0"
I20260806 09:16:14.382366 2084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:14.388378 2084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:14.392848 2084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:14.393155 2084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:14.412838 2084 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41559
I20260806 09:16:14.412883 2184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41559 every 8 connection(s)
I20260806 09:16:14.414273 2084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:14.415338 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 2084
I20260806 09:16:14.415545 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:14.416141 3348 external_mini_cluster.cc:1428] Setting key c00dbed7e6c4c5481104e14c49dedab0
I20260806 09:16:14.418903 2185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:14.423938 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:14.417961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:46775 (local address 127.0.0.1:41559)
0806 09:16:14.418780 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:14.418793 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:14.418836 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:16:14.420545 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:14.420565 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:14.420879 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:14.421205 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:14.422247 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:14.422253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:14.422264 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:14.422278 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:14.422356 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:14.422380 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:14.422631 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:14.422738 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 09:16:14.422970 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":370,"threads_started":1}
I20260806 09:16:14.427544 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:14.430356 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628: Bootstrap starting.
I20260806 09:16:14.434219 2185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:14.435227 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:14.432158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:58569 (local address 127.0.0.1:41559)
0806 09:16:14.432345 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:14.432350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:14.432893 (+ 543us) server_negotiation.cc:407] Connection header received
0806 09:16:14.433121 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:14.433126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:14.433195 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:14.433283 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:14.434498 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:14.434502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:14.434505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:14.434520 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:14.434564 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:14.434579 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:14.434684 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:14.434838 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 09:16:14.434886 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 09:16:14.435650 2185 log.cc:824] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:14.438544 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628: No bootstrap required, opened a new log
I20260806 09:16:14.446147 2185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93519348a4ae46efab86c1cb4ac5a628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260806 09:16:14.446581 2185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:14.446717 2185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93519348a4ae46efab86c1cb4ac5a628, State: Initialized, Role: FOLLOWER
I20260806 09:16:14.447307 2185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93519348a4ae46efab86c1cb4ac5a628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260806 09:16:14.447541 2185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:14.447690 2185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:14.447853 2185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:14.450207 2185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93519348a4ae46efab86c1cb4ac5a628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260806 09:16:14.450677 2185 leader_election.cc:302] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93519348a4ae46efab86c1cb4ac5a628; no voters:
I20260806 09:16:14.451220 2185 leader_election.cc:288] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:14.451326 2202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:14.452121 2202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [term 1 LEADER]: Becoming Leader. State: Replica: 93519348a4ae46efab86c1cb4ac5a628, State: Running, Role: LEADER
I20260806 09:16:14.452685 2202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93519348a4ae46efab86c1cb4ac5a628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260806 09:16:14.453406 2185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:14.455955 2204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93519348a4ae46efab86c1cb4ac5a628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93519348a4ae46efab86c1cb4ac5a628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } } }
I20260806 09:16:14.455926 2205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93519348a4ae46efab86c1cb4ac5a628. Latest consensus state: current_term: 1 leader_uuid: "93519348a4ae46efab86c1cb4ac5a628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93519348a4ae46efab86c1cb4ac5a628" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } } }
I20260806 09:16:14.456408 2204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:14.456435 2205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:14.467818 2212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:14.472585 2212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:14.481374 2212 catalog_manager.cc:1347] Generated new cluster ID: e7c17b13ed0e499f823e51b1f561a057
I20260806 09:16:14.481487 2212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:14.504664 2212 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:14.505177 2212 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:14.516728 2212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93519348a4ae46efab86c1cb4ac5a628: Generated new TSK 0
I20260806 09:16:14.517760 2212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:14.528191 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2084
W20260806 09:16:14.529496 2178 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.75:46775
W20260806 09:16:14.530193 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2084: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:14.530351 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 2084
2026-08-06T09:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:14 dist-test-slave-w4v5 krb5kdc[2236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:14 dist-test-slave-w4v5 krb5kdc[2236](info): set up 2 sockets
Aug 06 09:16:14 dist-test-slave-w4v5 krb5kdc[2236](info): commencing operation
krb5kdc: starting...
W20260806 09:16:15.856279 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:16 dist-test-slave-w4v5 krb5kdc[2236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:16.192412 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:16.566525 2254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:16.566834 2254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:16.566998 2254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:16.574370 2254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:16.574486 2254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:16.574579 2254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:16.574870 2254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:16.574954 2254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:16.585194 2254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:16.586547 2254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:16.587927 2254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:16.599977 2266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:16.599944 2270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:16.600014 2267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:16.600817 2254 server_base.cc:1034] running on GCE node
I20260806 09:16:16.601619 2254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:16.602990 2254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:16.604203 2254 hybrid_clock.cc:648] HybridClock initialized: now 1786007776604149 us; error 52 us; skew 500 ppm
I20260806 09:16:16.604561 2254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:16 dist-test-slave-w4v5 krb5kdc[2236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:16.607712 2254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:16.609166 2254 webserver.cc:460] Webserver started at http://127.0.0.1:33357/ using document root <none> and password file <none>
I20260806 09:16:16.609838 2254 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:16.609987 2254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:16.610297 2254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:16.613394 2254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "211a35d1f9a6458491ebf2f1a54e9f2f"
format_stamp: "Formatted at 2026-08-06 09:16:16 on dist-test-slave-w4v5"
server_key: "37ef7d18df3415e25bcf3ce309d1df01"
server_key_iv: "dba02774466755e02c5545d215224ee7"
server_key_version: "encryptionkey@0"
I20260806 09:16:16.614217 2254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "211a35d1f9a6458491ebf2f1a54e9f2f"
format_stamp: "Formatted at 2026-08-06 09:16:16 on dist-test-slave-w4v5"
server_key: "37ef7d18df3415e25bcf3ce309d1df01"
server_key_iv: "dba02774466755e02c5545d215224ee7"
server_key_version: "encryptionkey@0"
I20260806 09:16:16.619494 2254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 09:16:16.622941 2280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:16.624300 2254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:16:16.624480 2254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "211a35d1f9a6458491ebf2f1a54e9f2f"
format_stamp: "Formatted at 2026-08-06 09:16:16 on dist-test-slave-w4v5"
server_key: "37ef7d18df3415e25bcf3ce309d1df01"
server_key_iv: "dba02774466755e02c5545d215224ee7"
server_key_version: "encryptionkey@0"
I20260806 09:16:16.624677 2254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:16.637347 2254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:16.641342 2254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:16.641631 2254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:16.660645 2254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38225
I20260806 09:16:16.660682 2350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38225 every 8 connection(s)
I20260806 09:16:16.662014 2254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:16.665720 2352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:16.667017 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 2254
I20260806 09:16:16.667236 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:16.667701 3348 external_mini_cluster.cc:1428] Setting key 1dc55732f51e3fc871e516c923fbf52b
I20260806 09:16:16.679056 2352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f: Bootstrap starting.
Aug 06 09:16:16 dist-test-slave-w4v5 krb5kdc[2236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:16:16.682833 2352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:16.684068 2352 log.cc:824] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:16.687652 2352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f: No bootstrap required, opened a new log
I20260806 09:16:16.692606 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:16.669577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:44293 (local address 127.0.0.1:38225)
0806 09:16:16.670361 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:16.670373 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:16.670416 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:16:16.672157 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:16.672180 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:16.672576 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:16.672973 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:16.674568 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:16.676067 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:16.677705 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:16.678392 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:16.678599 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:16.683743 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:16.683752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:16.683770 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:16.683809 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:16.687402 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:16.688577 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:16.688585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:16.688594 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:16.688823 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:16.689427 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:16.689432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:16.689435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:16.690213 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:16.690568 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:16.691237 (+ 669us) server_negotiation.cc:299] Negotiation successful
0806 09:16:16.691591 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":348,"threads_started":1}
I20260806 09:16:16.696544 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:16.699433 2352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211a35d1f9a6458491ebf2f1a54e9f2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } }
I20260806 09:16:16.699961 2352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:16.700099 2352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211a35d1f9a6458491ebf2f1a54e9f2f, State: Initialized, Role: FOLLOWER
I20260806 09:16:16.700951 2352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211a35d1f9a6458491ebf2f1a54e9f2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } }
I20260806 09:16:16.701308 2352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:16.701471 2352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:16.701675 2352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:16.705569 2352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211a35d1f9a6458491ebf2f1a54e9f2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } }
I20260806 09:16:16.706262 2352 leader_election.cc:302] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211a35d1f9a6458491ebf2f1a54e9f2f; no voters:
I20260806 09:16:16.707022 2352 leader_election.cc:288] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:16.707206 2370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:16.708938 2370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [term 1 LEADER]: Becoming Leader. State: Replica: 211a35d1f9a6458491ebf2f1a54e9f2f, State: Running, Role: LEADER
I20260806 09:16:16.709882 2352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:16.709751 2370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211a35d1f9a6458491ebf2f1a54e9f2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } }
I20260806 09:16:16.719192 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:16.702509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:46391 (local address 127.0.0.1:38225)
0806 09:16:16.702729 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:16.702737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:16.703368 (+ 631us) server_negotiation.cc:407] Connection header received
0806 09:16:16.703544 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:16.703549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:16.703637 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:16.703736 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:16.705966 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:16.706834 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:16.708695 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:16.709411 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:16.709556 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:16.712856 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:16.712861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:16.712864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:16.712896 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:16.716079 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:16.716849 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:16.716853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:16.716855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:16.716994 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:16.717476 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:16.717480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:16.717482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:16.717852 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:16.718156 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:16.718579 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 09:16:16.718758 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:16:16.723523 2373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211a35d1f9a6458491ebf2f1a54e9f2f. Latest consensus state: current_term: 1 leader_uuid: "211a35d1f9a6458491ebf2f1a54e9f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211a35d1f9a6458491ebf2f1a54e9f2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } } }
I20260806 09:16:16.723475 2371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211a35d1f9a6458491ebf2f1a54e9f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211a35d1f9a6458491ebf2f1a54e9f2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } } }
I20260806 09:16:16.724022 2373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:16.724086 2371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:16.725181 2390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:16.729933 2390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:16.738570 2390 catalog_manager.cc:1347] Generated new cluster ID: 1595b611b28f4834a6e9cd9b37766965
I20260806 09:16:16.738694 2390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:16.745854 2390 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:16.747740 2390 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:16.758000 2390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 211a35d1f9a6458491ebf2f1a54e9f2f: Generated new TSK 0
I20260806 09:16:16.759053 2390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:16.766851 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2254
I20260806 09:16:16.768441 2345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:16.858348 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 2254
W20260806 09:16:16.884464 2233 connection.cc:535] client connection to 127.0.0.1:38225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38225): Connection reset by peer (error 104)
2026-08-06T09:16:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:16 dist-test-slave-w4v5 krb5kdc[2402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:16 dist-test-slave-w4v5 krb5kdc[2402](info): set up 2 sockets
Aug 06 09:16:16 dist-test-slave-w4v5 krb5kdc[2402](info): commencing operation
krb5kdc: starting...
W20260806 09:16:18.201195 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:18 dist-test-slave-w4v5 krb5kdc[2402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:18.526029 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:18.884972 2422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:18.885278 2422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:18.885375 2422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:18.892695 2422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:18.892804 2422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:18.892851 2422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:18.892915 2422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:18.892958 2422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:18.902877 2422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:18.904356 2422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:18.905627 2422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:18.917773 2439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:18.917961 2431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:18.917829 2432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:18.918578 2422 server_base.cc:1034] running on GCE node
I20260806 09:16:18.919495 2422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:18.920886 2422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:18.922124 2422 hybrid_clock.cc:648] HybridClock initialized: now 1786007778922086 us; error 51 us; skew 500 ppm
I20260806 09:16:18.922477 2422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:18 dist-test-slave-w4v5 krb5kdc[2402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:18.930413 2422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:18.932056 2422 webserver.cc:460] Webserver started at http://127.0.0.1:36743/ using document root <none> and password file <none>
I20260806 09:16:18.932749 2422 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:18.932901 2422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:18.933210 2422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:18.936231 2422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "95ddafee4dfb4473a76fca868207a381"
format_stamp: "Formatted at 2026-08-06 09:16:18 on dist-test-slave-w4v5"
server_key: "5c67078d6334bf5943d9784466b2424b"
server_key_iv: "67de593217d5148c170b6bc1d70e48bd"
server_key_version: "encryptionkey@0"
I20260806 09:16:18.937009 2422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "95ddafee4dfb4473a76fca868207a381"
format_stamp: "Formatted at 2026-08-06 09:16:18 on dist-test-slave-w4v5"
server_key: "5c67078d6334bf5943d9784466b2424b"
server_key_iv: "67de593217d5148c170b6bc1d70e48bd"
server_key_version: "encryptionkey@0"
I20260806 09:16:18.942167 2422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:16:18.945556 2448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:18.946833 2422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:16:18.947016 2422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "95ddafee4dfb4473a76fca868207a381"
format_stamp: "Formatted at 2026-08-06 09:16:18 on dist-test-slave-w4v5"
server_key: "5c67078d6334bf5943d9784466b2424b"
server_key_iv: "67de593217d5148c170b6bc1d70e48bd"
server_key_version: "encryptionkey@0"
I20260806 09:16:18.947263 2422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:18.973172 2422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:18.977561 2422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:18.977865 2422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:18.997026 2422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40497
I20260806 09:16:18.997074 2529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40497 every 8 connection(s)
I20260806 09:16:18.998409 2422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:18.999558 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 2422
I20260806 09:16:18.999763 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:19.000213 3348 external_mini_cluster.cc:1428] Setting key 764d2da7491e957369f3526e4c986861
I20260806 09:16:19.002707 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:19.013389 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:19.001931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:33247 (local address 127.0.0.1:40497)
0806 09:16:19.002727 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:19.002738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:19.002779 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:16:19.004457 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:19.004477 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:19.004844 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:19.005161 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:19.005767 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:19.007050 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:19.008985 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:19.009599 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:19.009795 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:19.011060 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:19.011068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:19.011125 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:19.011709 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:19.011784 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:19.011811 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:19.011936 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:19.012226 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 09:16:19.012437 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":361,"threads_started":1}
I20260806 09:16:19.015199 2532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381: Bootstrap starting.
I20260806 09:16:19.017509 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:19.018496 2532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:19.019549 2532 log.cc:824] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:19.022439 2532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381: No bootstrap required, opened a new log
I20260806 09:16:19.029198 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:19.022108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:50527 (local address 127.0.0.1:40497)
0806 09:16:19.022284 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:19.022290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:19.022702 (+ 412us) server_negotiation.cc:407] Connection header received
0806 09:16:19.022861 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:19.022868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:19.022975 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:19.023130 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:19.023755 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:19.024967 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:19.026401 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:19.027096 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:19.027234 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:19.028347 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:19.028353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:19.028358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:19.028385 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:19.028428 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:19.028449 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:19.028574 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:19.028739 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 09:16:19.028789 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:16:19.031769 2532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ddafee4dfb4473a76fca868207a381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260806 09:16:19.032161 2532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:19.032284 2532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ddafee4dfb4473a76fca868207a381, State: Initialized, Role: FOLLOWER
I20260806 09:16:19.032910 2532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ddafee4dfb4473a76fca868207a381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260806 09:16:19.033150 2532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:19.033299 2532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:19.033483 2532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:19.035782 2532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ddafee4dfb4473a76fca868207a381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260806 09:16:19.036307 2532 leader_election.cc:302] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95ddafee4dfb4473a76fca868207a381; no voters:
I20260806 09:16:19.036860 2532 leader_election.cc:288] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:19.037032 2549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:19.037797 2549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [term 1 LEADER]: Becoming Leader. State: Replica: 95ddafee4dfb4473a76fca868207a381, State: Running, Role: LEADER
I20260806 09:16:19.038445 2549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ddafee4dfb4473a76fca868207a381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260806 09:16:19.039197 2532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:19.041666 2550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95ddafee4dfb4473a76fca868207a381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ddafee4dfb4473a76fca868207a381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260806 09:16:19.041656 2552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95ddafee4dfb4473a76fca868207a381. Latest consensus state: current_term: 1 leader_uuid: "95ddafee4dfb4473a76fca868207a381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ddafee4dfb4473a76fca868207a381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260806 09:16:19.042199 2552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:19.042174 2550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:19.044129 2564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:19.049394 2564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:19.058526 2564 catalog_manager.cc:1347] Generated new cluster ID: fb02a4efd65543a0bc640808b9238455
I20260806 09:16:19.058642 2564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:19.073374 2564 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:19.074659 2564 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:19.086283 2564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95ddafee4dfb4473a76fca868207a381: Generated new TSK 0
I20260806 09:16:19.087159 2564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:19.096865 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2422
W20260806 09:16:19.098145 2523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.75:33247
W20260806 09:16:19.098929 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2422: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:19.099124 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 2422
W20260806 09:16:19.127408 2400 connection.cc:535] client connection to 127.0.0.1:40497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40497): Connection reset by peer (error 104)
2026-08-06T09:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:19 dist-test-slave-w4v5 krb5kdc[2586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:19 dist-test-slave-w4v5 krb5kdc[2586](info): set up 2 sockets
Aug 06 09:16:19 dist-test-slave-w4v5 krb5kdc[2586](info): commencing operation
krb5kdc: starting...
W20260806 09:16:20.424435 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:20 dist-test-slave-w4v5 krb5kdc[2586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:16:20.758400 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:21.104857 2608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:21.105152 2608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:21.105252 2608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:21.112421 2608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:21.112546 2608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:21.112596 2608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:21.112658 2608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:21.112704 2608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:21.122844 2608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:21.124189 2608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:21.125499 2608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:21.137307 2616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:21.137398 2615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:21.137398 2618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:21.138160 2608 server_base.cc:1034] running on GCE node
I20260806 09:16:21.138978 2608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:21.140332 2608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:21.141541 2608 hybrid_clock.cc:648] HybridClock initialized: now 1786007781141504 us; error 30 us; skew 500 ppm
I20260806 09:16:21.141892 2608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:21 dist-test-slave-w4v5 krb5kdc[2586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:21.145081 2608 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:16:21.146484 2608 webserver.cc:460] Webserver started at http://127.0.0.1:43145/ using document root <none> and password file <none>
I20260806 09:16:21.147202 2608 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:21.147336 2608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:21.147644 2608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:21.150606 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "f44743602aaf4187811519d9ea41957a"
format_stamp: "Formatted at 2026-08-06 09:16:21 on dist-test-slave-w4v5"
server_key: "f696c93afe01038ce6fe9f92a2088d6f"
server_key_iv: "d147fe4bd5ae7444b0ced3242141e22f"
server_key_version: "encryptionkey@0"
I20260806 09:16:21.151446 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "f44743602aaf4187811519d9ea41957a"
format_stamp: "Formatted at 2026-08-06 09:16:21 on dist-test-slave-w4v5"
server_key: "f696c93afe01038ce6fe9f92a2088d6f"
server_key_iv: "d147fe4bd5ae7444b0ced3242141e22f"
server_key_version: "encryptionkey@0"
I20260806 09:16:21.156435 2608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:16:21.159835 2630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:21.161098 2608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:16:21.161271 2608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "f44743602aaf4187811519d9ea41957a"
format_stamp: "Formatted at 2026-08-06 09:16:21 on dist-test-slave-w4v5"
server_key: "f696c93afe01038ce6fe9f92a2088d6f"
server_key_iv: "d147fe4bd5ae7444b0ced3242141e22f"
server_key_version: "encryptionkey@0"
I20260806 09:16:21.161460 2608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:21.167220 2608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:21.171193 2608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:21.171478 2608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:21.190704 2608 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46819
I20260806 09:16:21.190781 2719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46819 every 8 connection(s)
I20260806 09:16:21.192118 2608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:21.195920 2721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:21.200819 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 2608
I20260806 09:16:21.201040 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:21.201596 3348 external_mini_cluster.cc:1428] Setting key dcbce310d42b29a6ccd4b5b88822a745
I20260806 09:16:21.208215 2721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a: Bootstrap starting.
I20260806 09:16:21.210892 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:21.203609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:50073 (local address 127.0.0.1:46819)
0806 09:16:21.204468 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:21.204487 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:21.204539 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:16:21.206999 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:21.207026 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:21.207457 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:21.207887 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:21.208904 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:21.208911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:21.208927 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:21.208943 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:21.209029 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:21.209057 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:21.209270 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:21.209412 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 09:16:21.209604 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":342,"threads_started":1}
I20260806 09:16:21.212057 2721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:21.213135 2721 log.cc:824] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:21.214749 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:21.216573 2721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a: No bootstrap required, opened a new log
I20260806 09:16:21.222551 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:21.220115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:49075 (local address 127.0.0.1:46819)
0806 09:16:21.220295 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:21.220301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:21.220583 (+ 282us) server_negotiation.cc:407] Connection header received
0806 09:16:21.220718 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:21.220723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:21.220780 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:21.220898 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:21.221778 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:21.221785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:21.221789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:21.221813 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:21.221866 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:21.221885 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:21.222019 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:21.222101 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 09:16:21.222160 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:16:21.225991 2721 raft_consensus.cc:348] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44743602aaf4187811519d9ea41957a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260806 09:16:21.226342 2721 raft_consensus.cc:374] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:21.226416 2721 raft_consensus.cc:729] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44743602aaf4187811519d9ea41957a, State: Initialized, Role: FOLLOWER
I20260806 09:16:21.226915 2721 consensus_queue.cc:260] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44743602aaf4187811519d9ea41957a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260806 09:16:21.227183 2721 raft_consensus.cc:388] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:21.227293 2721 raft_consensus.cc:482] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:21.227420 2721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:21.229578 2721 raft_consensus.cc:504] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44743602aaf4187811519d9ea41957a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260806 09:16:21.230114 2721 leader_election.cc:302] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f44743602aaf4187811519d9ea41957a; no voters:
I20260806 09:16:21.230754 2721 leader_election.cc:288] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:21.230901 2733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:21.231607 2733 raft_consensus.cc:686] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [term 1 LEADER]: Becoming Leader. State: Replica: f44743602aaf4187811519d9ea41957a, State: Running, Role: LEADER
I20260806 09:16:21.232158 2733 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44743602aaf4187811519d9ea41957a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260806 09:16:21.232952 2721 sys_catalog.cc:564] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:21.235272 2735 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f44743602aaf4187811519d9ea41957a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44743602aaf4187811519d9ea41957a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } } }
I20260806 09:16:21.235319 2736 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f44743602aaf4187811519d9ea41957a. Latest consensus state: current_term: 1 leader_uuid: "f44743602aaf4187811519d9ea41957a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44743602aaf4187811519d9ea41957a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } } }
I20260806 09:16:21.235752 2736 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:21.235752 2735 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:21.237290 2742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:21.243170 2742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:21.252262 2742 catalog_manager.cc:1347] Generated new cluster ID: 9bbd70570e2a4b818598ab89e54899f8
I20260806 09:16:21.252409 2742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:21.267663 2742 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:21.268111 2742 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:21.280536 2742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f44743602aaf4187811519d9ea41957a: Generated new TSK 0
I20260806 09:16:21.281390 2742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:21.289307 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2608
W20260806 09:16:21.290436 2709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.75:50073
W20260806 09:16:21.291230 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2608: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:21.291383 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 2608
2026-08-06T09:16:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:21 dist-test-slave-w4v5 krb5kdc[2777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:21 dist-test-slave-w4v5 krb5kdc[2777](info): set up 2 sockets
Aug 06 09:16:21 dist-test-slave-w4v5 krb5kdc[2777](info): commencing operation
krb5kdc: starting...
W20260806 09:16:22.618255 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:22 dist-test-slave-w4v5 krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:16:22.956279 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40981
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:40981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:23.303507 2796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:23.303813 2796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:23.304006 2796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:23.311322 2796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:23.311472 2796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:23.311578 2796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:23.311685 2796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:23.311749 2796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:23.321617 2796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:40981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:40981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:23.322983 2796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:23.324378 2796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:23.336807 2804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:23.336840 2806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:23.336885 2808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:23.338048 2796 server_base.cc:1034] running on GCE node
I20260806 09:16:23.338831 2796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:23.340253 2796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:23.341452 2796 hybrid_clock.cc:648] HybridClock initialized: now 1786007783341411 us; error 38 us; skew 500 ppm
I20260806 09:16:23.341800 2796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:23 dist-test-slave-w4v5 krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007783, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:23.345005 2796 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:16:23.346482 2796 webserver.cc:460] Webserver started at http://127.3.69.62:36947/ using document root <none> and password file <none>
I20260806 09:16:23.347208 2796 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:23.347322 2796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:23.347653 2796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:23.350636 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "3ba16e69fb6f4d54babe28aed0582da1"
format_stamp: "Formatted at 2026-08-06 09:16:23 on dist-test-slave-w4v5"
server_key: "9f2f56a176e1163946786f909bd49773"
server_key_iv: "1116da2e5a8d7c384379686440479c89"
server_key_version: "encryptionkey@0"
I20260806 09:16:23.351424 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "3ba16e69fb6f4d54babe28aed0582da1"
format_stamp: "Formatted at 2026-08-06 09:16:23 on dist-test-slave-w4v5"
server_key: "9f2f56a176e1163946786f909bd49773"
server_key_iv: "1116da2e5a8d7c384379686440479c89"
server_key_version: "encryptionkey@0"
I20260806 09:16:23.356582 2796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:16:23.360154 2817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:23.361572 2796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:16:23.361730 2796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "3ba16e69fb6f4d54babe28aed0582da1"
format_stamp: "Formatted at 2026-08-06 09:16:23 on dist-test-slave-w4v5"
server_key: "9f2f56a176e1163946786f909bd49773"
server_key_iv: "1116da2e5a8d7c384379686440479c89"
server_key_version: "encryptionkey@0"
I20260806 09:16:23.361904 2796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:23.384759 2796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:23.389310 2796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:23.389559 2796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:23.408679 2796 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:40981
I20260806 09:16:23.408718 2897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:40981 every 8 connection(s)
I20260806 09:16:23.410064 2796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:23.413915 2898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:23.418915 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 2796
I20260806 09:16:23.419142 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:23.419577 3348 external_mini_cluster.cc:1428] Setting key b5057c8b5ccb3c136c5245bab1febd59
I20260806 09:16:23.424894 2898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1: Bootstrap starting.
I20260806 09:16:23.428679 2898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:23.429934 2898 log.cc:824] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:23.433537 2898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1: No bootstrap required, opened a new log
Aug 06 09:16:23 dist-test-slave-w4v5 krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:16:23.441617 2898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba16e69fb6f4d54babe28aed0582da1" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40981 } }
I20260806 09:16:23.441999 2898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:23.442131 2898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba16e69fb6f4d54babe28aed0582da1, State: Initialized, Role: FOLLOWER
I20260806 09:16:23.442770 2898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba16e69fb6f4d54babe28aed0582da1" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40981 } }
I20260806 09:16:23.443065 2898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:23.443303 2898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:23.443495 2898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:23.444342 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:23.421356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51549 (local address 127.3.69.62:40981)
0806 09:16:23.422332 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:23.422345 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:23.422392 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:16:23.424256 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:23.424279 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:23.424721 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:23.425102 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:23.427128 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:23.428480 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:23.430293 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:23.431460 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:23.431591 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:23.436644 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:23.436652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:23.436668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:23.436718 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:23.439962 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:23.440761 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:23.440768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:23.440774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:23.440942 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:23.441421 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:23.441426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:23.441428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:23.441967 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:23.442289 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:23.442806 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 09:16:23.443183 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":444,"threads_started":1}
I20260806 09:16:23.445979 2898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba16e69fb6f4d54babe28aed0582da1" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40981 } }
I20260806 09:16:23.446485 2898 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba16e69fb6f4d54babe28aed0582da1; no voters:
I20260806 09:16:23.447220 2898 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:23.447350 2909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:23.448328 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:23.448487 2909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba16e69fb6f4d54babe28aed0582da1, State: Running, Role: LEADER
I20260806 09:16:23.449079 2909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba16e69fb6f4d54babe28aed0582da1" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40981 } }
I20260806 09:16:23.449599 2898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:23.452004 2911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba16e69fb6f4d54babe28aed0582da1. Latest consensus state: current_term: 1 leader_uuid: "3ba16e69fb6f4d54babe28aed0582da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba16e69fb6f4d54babe28aed0582da1" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40981 } } }
I20260806 09:16:23.452509 2911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:23.454587 2910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba16e69fb6f4d54babe28aed0582da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba16e69fb6f4d54babe28aed0582da1" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 40981 } } }
I20260806 09:16:23.454984 2910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:23.467015 2925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:23.474226 2925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:23.495229 2944 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:16:23.497541 2925 catalog_manager.cc:1347] Generated new cluster ID: b72ff89addb04cc3b27b3852c2913167
I20260806 09:16:23.497651 2925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:23.505836 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:23.480819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55233 (local address 127.3.69.62:40981)
0806 09:16:23.481005 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:23.481011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:23.488345 (+ 7334us) server_negotiation.cc:407] Connection header received
0806 09:16:23.488582 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:23.488588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:23.488681 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:23.488807 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:23.491045 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:23.492295 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:23.494228 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:23.494980 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:23.495142 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:23.498727 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:23.498736 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:23.498743 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:23.498787 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:23.502510 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:23.503502 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:23.503507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:23.503511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:23.503673 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:23.504213 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:23.504217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:23.504221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:23.504574 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:23.504841 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:23.505279 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 09:16:23.505454 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:16:23.512457 2925 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:23.513762 2925 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:23.521512 2925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba16e69fb6f4d54babe28aed0582da1: Generated new TSK 0
I20260806 09:16:23.522361 2925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:23.532624 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2796
I20260806 09:16:23.533833 2889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:23.625963 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 2796
W20260806 09:16:23.654450 2771 connection.cc:535] client connection to 127.3.69.62:40981 recv error: Network error: failed to read from TLS socket (remote: 127.3.69.62:40981): Connection reset by peer (error 104)
2026-08-06T09:16:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:23 dist-test-slave-w4v5 krb5kdc[2953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:23 dist-test-slave-w4v5 krb5kdc[2953](info): set up 2 sockets
Aug 06 09:16:23 dist-test-slave-w4v5 krb5kdc[2953](info): commencing operation
krb5kdc: starting...
W20260806 09:16:24.965691 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:25 dist-test-slave-w4v5 krb5kdc[2953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:16:25.301828 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:38545
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:38545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:25.653997 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:25.654279 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:25.654445 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:25.661630 2980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:25.661806 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:25.661904 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:25.662014 2980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:25.662101 2980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:25.672214 2980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:38545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:38545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:25.673454 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:25.674798 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:25.686697 2987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:25.686784 2990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:25.686762 2993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:25.687893 2980 server_base.cc:1034] running on GCE node
I20260806 09:16:25.688668 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:25.690016 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:25.691185 2980 hybrid_clock.cc:648] HybridClock initialized: now 1786007785691121 us; error 48 us; skew 500 ppm
I20260806 09:16:25.691536 2980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:25 dist-test-slave-w4v5 krb5kdc[2953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007785, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:25.694916 2980 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:16:25.696435 2980 webserver.cc:460] Webserver started at http://127.3.69.62:40193/ using document root <none> and password file <none>
I20260806 09:16:25.697125 2980 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:25.697258 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:25.697594 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:25.700663 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "422495f7fe124c8e8ef7ab63aee1cb1b"
format_stamp: "Formatted at 2026-08-06 09:16:25 on dist-test-slave-w4v5"
server_key: "7978e495027ca345487a9be953069ffb"
server_key_iv: "e98712433b587ed8bcd3bf0ea4ce43b7"
server_key_version: "encryptionkey@0"
I20260806 09:16:25.701443 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "422495f7fe124c8e8ef7ab63aee1cb1b"
format_stamp: "Formatted at 2026-08-06 09:16:25 on dist-test-slave-w4v5"
server_key: "7978e495027ca345487a9be953069ffb"
server_key_iv: "e98712433b587ed8bcd3bf0ea4ce43b7"
server_key_version: "encryptionkey@0"
I20260806 09:16:25.706661 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 09:16:25.710136 3004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:25.711642 2980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:16:25.711833 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "422495f7fe124c8e8ef7ab63aee1cb1b"
format_stamp: "Formatted at 2026-08-06 09:16:25 on dist-test-slave-w4v5"
server_key: "7978e495027ca345487a9be953069ffb"
server_key_iv: "e98712433b587ed8bcd3bf0ea4ce43b7"
server_key_version: "encryptionkey@0"
I20260806 09:16:25.712035 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:25.727877 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:25.732355 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:25.732657 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:25.751870 2980 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:38545
I20260806 09:16:25.751909 3077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:38545 every 8 connection(s)
I20260806 09:16:25.753280 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:25.754961 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 2980
I20260806 09:16:25.755185 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:25.755766 3348 external_mini_cluster.cc:1428] Setting key 5352cebf2856896f6250b1c3792cb5d1
I20260806 09:16:25.759209 3078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:25.769264 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:25.757643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39759 (local address 127.3.69.62:38545)
0806 09:16:25.758351 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:25.758362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:25.758404 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:16:25.760175 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:25.760197 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:25.760544 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:25.760884 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:25.761465 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:25.762816 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:25.764744 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:25.765386 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:25.765511 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:25.766838 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:25.766845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:25.766858 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:25.767444 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:25.767527 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:25.767552 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:25.767692 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:25.768085 (+ 393us) server_negotiation.cc:299] Negotiation successful
0806 09:16:25.768269 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":283,"threads_started":1}
I20260806 09:16:25.771433 3078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b: Bootstrap starting.
I20260806 09:16:25.773020 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:25.774972 3078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:25.776292 3078 log.cc:824] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:25.779546 3078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b: No bootstrap required, opened a new log
I20260806 09:16:25.785573 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:25.778564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42323 (local address 127.3.69.62:38545)
0806 09:16:25.778763 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:25.778770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:25.779320 (+ 550us) server_negotiation.cc:407] Connection header received
0806 09:16:25.779479 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:25.779484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:25.779571 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:25.779696 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:25.780216 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:25.781332 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:25.782749 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:25.783489 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:25.783625 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:25.784782 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:25.784793 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:25.784798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:25.784824 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:25.784867 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:25.784886 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:25.784997 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:25.785179 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 09:16:25.785241 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:16:25.788447 3078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422495f7fe124c8e8ef7ab63aee1cb1b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38545 } }
I20260806 09:16:25.788842 3078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:25.788965 3078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422495f7fe124c8e8ef7ab63aee1cb1b, State: Initialized, Role: FOLLOWER
I20260806 09:16:25.789553 3078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422495f7fe124c8e8ef7ab63aee1cb1b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38545 } }
I20260806 09:16:25.789760 3078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:25.789858 3078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:25.789996 3078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:25.792196 3078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422495f7fe124c8e8ef7ab63aee1cb1b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38545 } }
I20260806 09:16:25.792634 3078 leader_election.cc:302] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 422495f7fe124c8e8ef7ab63aee1cb1b; no voters:
I20260806 09:16:25.793351 3078 leader_election.cc:288] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:25.793483 3098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:25.794210 3098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [term 1 LEADER]: Becoming Leader. State: Replica: 422495f7fe124c8e8ef7ab63aee1cb1b, State: Running, Role: LEADER
I20260806 09:16:25.794729 3098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422495f7fe124c8e8ef7ab63aee1cb1b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38545 } }
I20260806 09:16:25.795521 3078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:25.797622 3100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 422495f7fe124c8e8ef7ab63aee1cb1b. Latest consensus state: current_term: 1 leader_uuid: "422495f7fe124c8e8ef7ab63aee1cb1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422495f7fe124c8e8ef7ab63aee1cb1b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38545 } } }
I20260806 09:16:25.797647 3099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "422495f7fe124c8e8ef7ab63aee1cb1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "422495f7fe124c8e8ef7ab63aee1cb1b" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 38545 } } }
I20260806 09:16:25.798046 3100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:25.798105 3099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:25.801164 3111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:25.805410 3111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:25.814272 3111 catalog_manager.cc:1347] Generated new cluster ID: 952ee53ae8d648f2ab30524e69cf6037
I20260806 09:16:25.814405 3111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:25.832770 3111 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:25.834220 3111 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:25.843067 3111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 422495f7fe124c8e8ef7ab63aee1cb1b: Generated new TSK 0
I20260806 09:16:25.843905 3111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:25.852324 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2980
W20260806 09:16:25.853524 3072 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39759
W20260806 09:16:25.854306 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 2980: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:25.854462 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 2980
W20260806 09:16:25.883366 2951 connection.cc:535] client connection to 127.3.69.62:38545 recv error: Network error: failed to read from TLS socket (remote: 127.3.69.62:38545): Connection reset by peer (error 104)
2026-08-06T09:16:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:25 dist-test-slave-w4v5 krb5kdc[3134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:25 dist-test-slave-w4v5 krb5kdc[3134](info): set up 2 sockets
Aug 06 09:16:25 dist-test-slave-w4v5 krb5kdc[3134](info): commencing operation
krb5kdc: starting...
W20260806 09:16:27.199326 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:27 dist-test-slave-w4v5 krb5kdc[3134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:16:27.525863 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:33227
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:33227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:27.875893 3157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:27.876164 3157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:27.876256 3157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:27.883422 3157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:27.883538 3157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:27.883591 3157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:27.883656 3157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:27.883700 3157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:27.893376 3157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:33227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:33227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:27.894601 3157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:27.895962 3157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:27.908151 3166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:27.908133 3168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:27.908341 3163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:27.909113 3157 server_base.cc:1034] running on GCE node
I20260806 09:16:27.909956 3157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:27.911397 3157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:27.912612 3157 hybrid_clock.cc:648] HybridClock initialized: now 1786007787912538 us; error 66 us; skew 500 ppm
I20260806 09:16:27.912969 3157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:27 dist-test-slave-w4v5 krb5kdc[3134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007787, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:27.916308 3157 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:16:27.917891 3157 webserver.cc:460] Webserver started at http://127.3.69.62:35251/ using document root <none> and password file <none>
I20260806 09:16:27.918561 3157 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:27.918701 3157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:27.919021 3157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:27.922029 3157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "24763eac85f24f1fbf6f698a048dee92"
format_stamp: "Formatted at 2026-08-06 09:16:27 on dist-test-slave-w4v5"
server_key: "6ec9631e35761a180a34048efd6f0631"
server_key_iv: "99b31a77720876ba2ebc98e1408332d1"
server_key_version: "encryptionkey@0"
I20260806 09:16:27.922796 3157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "24763eac85f24f1fbf6f698a048dee92"
format_stamp: "Formatted at 2026-08-06 09:16:27 on dist-test-slave-w4v5"
server_key: "6ec9631e35761a180a34048efd6f0631"
server_key_iv: "99b31a77720876ba2ebc98e1408332d1"
server_key_version: "encryptionkey@0"
I20260806 09:16:27.927944 3157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:16:27.931273 3175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:27.932471 3157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:16:27.932655 3157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "24763eac85f24f1fbf6f698a048dee92"
format_stamp: "Formatted at 2026-08-06 09:16:27 on dist-test-slave-w4v5"
server_key: "6ec9631e35761a180a34048efd6f0631"
server_key_iv: "99b31a77720876ba2ebc98e1408332d1"
server_key_version: "encryptionkey@0"
I20260806 09:16:27.932864 3157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:27.938628 3157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:27.942553 3157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:27.942860 3157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:27.961918 3157 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:33227
I20260806 09:16:27.961961 3253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:33227 every 8 connection(s)
I20260806 09:16:27.963289 3157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:27.966850 3254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:27.967355 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3157
I20260806 09:16:27.967530 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:27.967969 3348 external_mini_cluster.cc:1428] Setting key 44e349341f5c3032201e2ea4d7452c1b
I20260806 09:16:27.977468 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:27.969742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47841 (local address 127.3.69.62:33227)
0806 09:16:27.970648 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:27.970663 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:27.970713 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:16:27.973809 (+ 3096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:27.973837 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:27.974273 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:27.974727 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:27.975670 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:27.975678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:27.975694 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:27.975709 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:27.975769 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:27.975791 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:27.975990 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:27.976116 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 09:16:27.976288 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":318,"threads_started":1}
I20260806 09:16:27.981034 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:27.981379 3254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92: Bootstrap starting.
I20260806 09:16:27.984396 3254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:27.985539 3254 log.cc:824] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:27.988111 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:27.985375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54735 (local address 127.3.69.62:33227)
0806 09:16:27.985583 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:27.985592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:27.986058 (+ 466us) server_negotiation.cc:407] Connection header received
0806 09:16:27.986256 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:27.986261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:27.986326 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:27.986442 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:27.987433 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:27.987438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:27.987443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:27.987460 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:27.987507 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:27.987528 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:27.987653 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:27.987752 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 09:16:27.987814 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:16:27.988742 3254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92: No bootstrap required, opened a new log
I20260806 09:16:27.995857 3254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24763eac85f24f1fbf6f698a048dee92" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33227 } }
I20260806 09:16:27.996232 3254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:27.996340 3254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24763eac85f24f1fbf6f698a048dee92, State: Initialized, Role: FOLLOWER
I20260806 09:16:27.996874 3254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24763eac85f24f1fbf6f698a048dee92" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33227 } }
I20260806 09:16:27.997126 3254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:27.997277 3254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:27.997452 3254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:27.999688 3254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24763eac85f24f1fbf6f698a048dee92" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33227 } }
I20260806 09:16:28.000201 3254 leader_election.cc:302] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24763eac85f24f1fbf6f698a048dee92; no voters:
I20260806 09:16:28.000801 3254 leader_election.cc:288] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:28.000933 3270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:28.001648 3270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [term 1 LEADER]: Becoming Leader. State: Replica: 24763eac85f24f1fbf6f698a048dee92, State: Running, Role: LEADER
I20260806 09:16:28.002206 3270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24763eac85f24f1fbf6f698a048dee92" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33227 } }
I20260806 09:16:28.002970 3254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:28.005162 3271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24763eac85f24f1fbf6f698a048dee92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24763eac85f24f1fbf6f698a048dee92" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33227 } } }
I20260806 09:16:28.005681 3271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:28.006395 3273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24763eac85f24f1fbf6f698a048dee92. Latest consensus state: current_term: 1 leader_uuid: "24763eac85f24f1fbf6f698a048dee92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24763eac85f24f1fbf6f698a048dee92" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 33227 } } }
I20260806 09:16:28.006752 3273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:28.008052 3283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:28.013054 3283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:28.022197 3283 catalog_manager.cc:1347] Generated new cluster ID: 9b5d5f91ea6e48169e174bba11283c49
I20260806 09:16:28.022310 3283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:28.045924 3283 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:28.046360 3283 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:28.067138 3283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24763eac85f24f1fbf6f698a048dee92: Generated new TSK 0
I20260806 09:16:28.067958 3283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:28.076730 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3157
W20260806 09:16:28.078147 3249 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47841
W20260806 09:16:28.078835 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3157: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:28.078987 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3157
2026-08-06T09:16:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:28 dist-test-slave-w4v5 krb5kdc[3305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:28 dist-test-slave-w4v5 krb5kdc[3305](info): set up 2 sockets
Aug 06 09:16:28 dist-test-slave-w4v5 krb5kdc[3305](info): commencing operation
krb5kdc: starting...
W20260806 09:16:29.402238 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.278s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:29 dist-test-slave-w4v5 krb5kdc[3305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:16:29.738134 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39983
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:39983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:30.081624 3322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:30.081943 3322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:30.082108 3322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:30.089200 3322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:30.089306 3322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:30.089395 3322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:30.089507 3322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:30.089586 3322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:30.099400 3322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:39983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:30.100739 3322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:30.102033 3322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:30.114189 3343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:30.114276 3341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:30.114244 3354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:30.115031 3322 server_base.cc:1034] running on GCE node
I20260806 09:16:30.115893 3322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:30.117369 3322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:30.118649 3322 hybrid_clock.cc:648] HybridClock initialized: now 1786007790118578 us; error 67 us; skew 500 ppm
I20260806 09:16:30.119202 3322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:30 dist-test-slave-w4v5 krb5kdc[3305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:30.125234 3322 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:16:30.126839 3322 webserver.cc:460] Webserver started at http://127.3.69.62:42641/ using document root <none> and password file <none>
I20260806 09:16:30.127614 3322 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:30.127774 3322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:30.128083 3322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:30.131685 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "968077147655453e909a7079d8b94b99"
format_stamp: "Formatted at 2026-08-06 09:16:30 on dist-test-slave-w4v5"
server_key: "759ab1c56db5898994cde28b562a565f"
server_key_iv: "91616299dbf1bf820aa86ceb5edd9088"
server_key_version: "encryptionkey@0"
I20260806 09:16:30.132462 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "968077147655453e909a7079d8b94b99"
format_stamp: "Formatted at 2026-08-06 09:16:30 on dist-test-slave-w4v5"
server_key: "759ab1c56db5898994cde28b562a565f"
server_key_iv: "91616299dbf1bf820aa86ceb5edd9088"
server_key_version: "encryptionkey@0"
I20260806 09:16:30.137645 3322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:16:30.141099 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:30.142406 3322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:16:30.142601 3322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "968077147655453e909a7079d8b94b99"
format_stamp: "Formatted at 2026-08-06 09:16:30 on dist-test-slave-w4v5"
server_key: "759ab1c56db5898994cde28b562a565f"
server_key_iv: "91616299dbf1bf820aa86ceb5edd9088"
server_key_version: "encryptionkey@0"
I20260806 09:16:30.142802 3322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:30.157708 3322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:30.161612 3322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:30.161914 3322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:30.180871 3322 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:39983
I20260806 09:16:30.180892 3432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:39983 every 8 connection(s)
I20260806 09:16:30.182304 3322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:30.185997 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:30.191088 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3322
I20260806 09:16:30.191280 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:30.191924 3348 external_mini_cluster.cc:1428] Setting key 5fb09bef479fa3a3bee7c8a17c007c75
I20260806 09:16:30.197398 3434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99: Bootstrap starting.
I20260806 09:16:30.200485 3434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:30.201471 3434 log.cc:824] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:30.205017 3434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99: No bootstrap required, opened a new log
Aug 06 09:16:30 dist-test-slave-w4v5 krb5kdc[3305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:16:30.215868 3434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968077147655453e909a7079d8b94b99" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39983 } }
I20260806 09:16:30.216296 3434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:30.216394 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:30.193744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50569 (local address 127.3.69.62:39983)
0806 09:16:30.194545 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:30.194555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:30.194594 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:16:30.196392 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:30.196415 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:30.196778 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:30.197153 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:30.198629 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:30.199993 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:30.201412 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:30.202424 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:30.202628 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:30.207759 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:30.207771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:30.207789 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:30.207837 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:30.212002 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:30.212643 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:30.212653 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:30.212661 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:30.212861 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:30.213200 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:30.213206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:30.213209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:30.213966 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:30.214298 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:30.214569 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 09:16:30.215014 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":354,"threads_started":1}
I20260806 09:16:30.216436 3434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968077147655453e909a7079d8b94b99, State: Initialized, Role: FOLLOWER
I20260806 09:16:30.217121 3434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968077147655453e909a7079d8b94b99" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39983 } }
I20260806 09:16:30.217410 3434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:30.217587 3434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:30.217789 3434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:30.220284 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:30.220669 3434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968077147655453e909a7079d8b94b99" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39983 } }
I20260806 09:16:30.221148 3434 leader_election.cc:302] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 968077147655453e909a7079d8b94b99; no voters:
I20260806 09:16:30.221767 3434 leader_election.cc:288] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:30.221930 3439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:30.223841 3434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:30.229038 3439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [term 1 LEADER]: Becoming Leader. State: Replica: 968077147655453e909a7079d8b94b99, State: Running, Role: LEADER
I20260806 09:16:30.229683 3439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968077147655453e909a7079d8b94b99" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39983 } }
I20260806 09:16:30.233033 3441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 968077147655453e909a7079d8b94b99. Latest consensus state: current_term: 1 leader_uuid: "968077147655453e909a7079d8b94b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968077147655453e909a7079d8b94b99" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39983 } } }
I20260806 09:16:30.233448 3441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:30.233970 3440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "968077147655453e909a7079d8b94b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968077147655453e909a7079d8b94b99" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39983 } } }
I20260806 09:16:30.234308 3440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99 [sys.catalog]: This master's current role is: LEADER
W20260806 09:16:30.239683 3464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:16:30.239822 3464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:16:30.239974 3465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:30.241731 3465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:30.251448 3465 catalog_manager.cc:1347] Generated new cluster ID: 7a7ea452f46f440caec64129c1da6ffc
I20260806 09:16:30.251570 3465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:30.255409 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:30.231224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32989 (local address 127.3.69.62:39983)
0806 09:16:30.231421 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:30.231427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:30.236668 (+ 5241us) server_negotiation.cc:407] Connection header received
0806 09:16:30.236880 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:30.236886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:30.236971 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:30.237108 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:30.239330 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:30.240440 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:30.241633 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:30.242345 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:30.242515 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:30.244899 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:30.244904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:30.244912 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:30.244945 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:30.248368 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:30.249230 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:30.249236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:30.249240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:30.249412 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:30.249871 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:30.249876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:30.249879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:30.250308 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:30.250573 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:30.250988 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 09:16:30.251247 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:16:30.272059 3465 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:30.273609 3465 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:30.285154 3465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 968077147655453e909a7079d8b94b99: Generated new TSK 0
I20260806 09:16:30.286020 3465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:30.296175 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3322
I20260806 09:16:30.297365 3428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:30.390820 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3322
2026-08-06T09:16:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:30 dist-test-slave-w4v5 krb5kdc[3477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:30 dist-test-slave-w4v5 krb5kdc[3477](info): set up 2 sockets
Aug 06 09:16:30 dist-test-slave-w4v5 krb5kdc[3477](info): commencing operation
krb5kdc: starting...
W20260806 09:16:31.721629 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:31 dist-test-slave-w4v5 krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:16:32.075431 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37387
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:37387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:32.430394 3498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:32.430703 3498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:32.430799 3498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:32.438005 3498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:32.438117 3498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:32.438165 3498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:32.438221 3498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:32.438259 3498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:32.447976 3498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:37387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:37387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:32.449195 3498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:32.450428 3498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:32.462699 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:32.462674 3515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:32.468111 3510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:32.469018 3498 server_base.cc:1034] running on GCE node
I20260806 09:16:32.469911 3498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:32.471417 3498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:32.472622 3498 hybrid_clock.cc:648] HybridClock initialized: now 1786007792472576 us; error 34 us; skew 500 ppm
I20260806 09:16:32.472975 3498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:32 dist-test-slave-w4v5 krb5kdc[3477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:32.476151 3498 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:16:32.477763 3498 webserver.cc:460] Webserver started at http://127.3.69.62:42973/ using document root <none> and password file <none>
I20260806 09:16:32.478469 3498 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:32.478621 3498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:32.478925 3498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:32.481935 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "e8f346373b0e4764b07dd266759fca2a"
format_stamp: "Formatted at 2026-08-06 09:16:32 on dist-test-slave-w4v5"
server_key: "a428bd363287915307de53de2b495853"
server_key_iv: "f4383395db2de3407e98296071e1bcde"
server_key_version: "encryptionkey@0"
I20260806 09:16:32.482712 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "e8f346373b0e4764b07dd266759fca2a"
format_stamp: "Formatted at 2026-08-06 09:16:32 on dist-test-slave-w4v5"
server_key: "a428bd363287915307de53de2b495853"
server_key_iv: "f4383395db2de3407e98296071e1bcde"
server_key_version: "encryptionkey@0"
I20260806 09:16:32.488126 3498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 09:16:32.491495 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:32.492915 3498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 09:16:32.493125 3498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "e8f346373b0e4764b07dd266759fca2a"
format_stamp: "Formatted at 2026-08-06 09:16:32 on dist-test-slave-w4v5"
server_key: "a428bd363287915307de53de2b495853"
server_key_iv: "f4383395db2de3407e98296071e1bcde"
server_key_version: "encryptionkey@0"
I20260806 09:16:32.493330 3498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:32.505177 3498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:32.509377 3498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:32.509665 3498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:32.529196 3498 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:37387
I20260806 09:16:32.529220 3605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:37387 every 8 connection(s)
I20260806 09:16:32.530576 3498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:32.531018 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3498
I20260806 09:16:32.531235 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:32.531884 3348 external_mini_cluster.cc:1428] Setting key 8e02971c18adbb792df479f401637279
I20260806 09:16:32.534940 3606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:32.545608 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:32.533821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57505 (local address 127.3.69.62:37387)
0806 09:16:32.534669 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:32.534681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:32.534721 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:16:32.536415 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:32.536437 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:32.536816 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:32.537126 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:32.537784 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:32.539127 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:32.541128 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:32.541804 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:32.541931 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:32.543266 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:32.543274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:32.543286 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:32.543887 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:32.543958 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:32.543985 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:32.544075 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:32.544346 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 09:16:32.544657 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":390,"threads_started":1}
I20260806 09:16:32.547376 3606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a: Bootstrap starting.
I20260806 09:16:32.549461 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:32.550843 3606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:32.551882 3606 log.cc:824] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:32.554991 3606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a: No bootstrap required, opened a new log
I20260806 09:16:32.559911 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:32.553818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49451 (local address 127.3.69.62:37387)
0806 09:16:32.554008 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:32.554016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:32.554426 (+ 410us) server_negotiation.cc:407] Connection header received
0806 09:16:32.554648 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:32.554651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:32.554706 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:32.554809 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:32.555371 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:32.556090 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:32.557384 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:32.557859 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:32.558013 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:32.559130 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:32.559137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:32.559142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:32.559168 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:32.559205 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:32.559220 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:32.559343 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:32.559489 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 09:16:32.559582 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 09:16:32.563051 3606 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f346373b0e4764b07dd266759fca2a" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37387 } }
I20260806 09:16:32.563472 3606 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:32.563582 3606 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f346373b0e4764b07dd266759fca2a, State: Initialized, Role: FOLLOWER
I20260806 09:16:32.564248 3606 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f346373b0e4764b07dd266759fca2a" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37387 } }
I20260806 09:16:32.564524 3606 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:32.564707 3606 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:32.564937 3606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:32.567461 3606 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f346373b0e4764b07dd266759fca2a" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37387 } }
I20260806 09:16:32.568087 3606 leader_election.cc:302] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f346373b0e4764b07dd266759fca2a; no voters:
I20260806 09:16:32.568693 3606 leader_election.cc:288] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:32.568840 3619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:32.569550 3619 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [term 1 LEADER]: Becoming Leader. State: Replica: e8f346373b0e4764b07dd266759fca2a, State: Running, Role: LEADER
I20260806 09:16:32.570070 3619 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f346373b0e4764b07dd266759fca2a" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37387 } }
I20260806 09:16:32.570798 3606 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:32.573091 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f346373b0e4764b07dd266759fca2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f346373b0e4764b07dd266759fca2a" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37387 } } }
I20260806 09:16:32.573485 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:32.574266 3622 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f346373b0e4764b07dd266759fca2a. Latest consensus state: current_term: 1 leader_uuid: "e8f346373b0e4764b07dd266759fca2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f346373b0e4764b07dd266759fca2a" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 37387 } } }
I20260806 09:16:32.574594 3622 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:32.574697 3629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:32.581547 3629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:32.591408 3629 catalog_manager.cc:1347] Generated new cluster ID: e6dba2ccb2f74b9782cbbe87ceef384d
I20260806 09:16:32.591529 3629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:32.617846 3629 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:32.619216 3629 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:32.634593 3629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f346373b0e4764b07dd266759fca2a: Generated new TSK 0
I20260806 09:16:32.635479 3629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:32.645615 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3498
W20260806 09:16:32.646790 3598 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57505
W20260806 09:16:32.647492 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3498: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:32.647656 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3498
2026-08-06T09:16:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:32 dist-test-slave-w4v5 krb5kdc[3654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:32 dist-test-slave-w4v5 krb5kdc[3654](info): set up 2 sockets
Aug 06 09:16:32 dist-test-slave-w4v5 krb5kdc[3654](info): commencing operation
krb5kdc: starting...
W20260806 09:16:33.987658 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:34 dist-test-slave-w4v5 krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:16:34.328286 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42227
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:42227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:34.679610 3675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:34.679883 3675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:34.680078 3675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:34.687322 3675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:34.687476 3675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:34.687644 3675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:34.687762 3675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:34.687851 3675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:34.698971 3675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:42227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:42227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:34.700383 3675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:34.701870 3675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:34.714191 3686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:34.714270 3681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:34.714461 3680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:34.715370 3675 server_base.cc:1034] running on GCE node
I20260806 09:16:34.716161 3675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:34.717501 3675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:34.718758 3675 hybrid_clock.cc:648] HybridClock initialized: now 1786007794718710 us; error 37 us; skew 500 ppm
I20260806 09:16:34.719192 3675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:34 dist-test-slave-w4v5 krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:34.722594 3675 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:16:34.724123 3675 webserver.cc:460] Webserver started at http://127.3.69.62:34619/ using document root <none> and password file <none>
I20260806 09:16:34.724820 3675 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:34.724972 3675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:34.725294 3675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:34.728319 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "57ba8e7312d14926a0fbb27e427f74bd"
format_stamp: "Formatted at 2026-08-06 09:16:34 on dist-test-slave-w4v5"
server_key: "4810c7dfc8c2d0c197c429d201b86ea0"
server_key_iv: "5d453697afa1d7cf78dcd773301f2ad8"
server_key_version: "encryptionkey@0"
I20260806 09:16:34.729113 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "57ba8e7312d14926a0fbb27e427f74bd"
format_stamp: "Formatted at 2026-08-06 09:16:34 on dist-test-slave-w4v5"
server_key: "4810c7dfc8c2d0c197c429d201b86ea0"
server_key_iv: "5d453697afa1d7cf78dcd773301f2ad8"
server_key_version: "encryptionkey@0"
I20260806 09:16:34.734725 3675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:16:34.738190 3695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:34.739573 3675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 09:16:34.739774 3675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "57ba8e7312d14926a0fbb27e427f74bd"
format_stamp: "Formatted at 2026-08-06 09:16:34 on dist-test-slave-w4v5"
server_key: "4810c7dfc8c2d0c197c429d201b86ea0"
server_key_iv: "5d453697afa1d7cf78dcd773301f2ad8"
server_key_version: "encryptionkey@0"
I20260806 09:16:34.739975 3675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:34.745712 3675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:34.749943 3675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:34.750250 3675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:34.773451 3675 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:42227
I20260806 09:16:34.773562 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:42227 every 8 connection(s)
I20260806 09:16:34.774881 3675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:34.778646 3775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:34.780683 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3675
I20260806 09:16:34.780879 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:34.781435 3348 external_mini_cluster.cc:1428] Setting key 623aedf5e2e8faebbdee03f82b92448a
I20260806 09:16:34.789182 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:34.783360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48263 (local address 127.3.69.62:42227)
0806 09:16:34.784157 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:34.784172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:34.784214 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:16:34.785772 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:34.785793 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:34.786132 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:34.786509 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:34.787655 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:34.787662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:34.787674 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:34.787688 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:34.787744 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:34.787763 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:34.788010 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:34.788154 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 09:16:34.788322 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":358,"threads_started":1}
I20260806 09:16:34.792491 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd: Bootstrap starting.
I20260806 09:16:34.792905 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:34.796183 3775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:34.797350 3775 log.cc:824] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:34.800530 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:34.797778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53549 (local address 127.3.69.62:42227)
0806 09:16:34.797995 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:34.798001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:34.798465 (+ 464us) server_negotiation.cc:407] Connection header received
0806 09:16:34.798610 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:34.798618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:34.798687 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:34.798814 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:34.799807 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:34.799811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:34.799814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:16:34.799829 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:34.799866 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:16:34.799881 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:34.799975 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:34.800169 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 09:16:34.800215 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:16:34.801226 3775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd: No bootstrap required, opened a new log
I20260806 09:16:34.809652 3775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ba8e7312d14926a0fbb27e427f74bd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42227 } }
I20260806 09:16:34.810086 3775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:16:34.810220 3775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ba8e7312d14926a0fbb27e427f74bd, State: Initialized, Role: FOLLOWER
I20260806 09:16:34.810832 3775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ba8e7312d14926a0fbb27e427f74bd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42227 } }
I20260806 09:16:34.811141 3775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:16:34.811319 3775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:16:34.811503 3775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:34.813956 3775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ba8e7312d14926a0fbb27e427f74bd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42227 } }
I20260806 09:16:34.814505 3775 leader_election.cc:302] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57ba8e7312d14926a0fbb27e427f74bd; no voters:
I20260806 09:16:34.815161 3775 leader_election.cc:288] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:16:34.815297 3787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:34.816166 3787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [term 1 LEADER]: Becoming Leader. State: Replica: 57ba8e7312d14926a0fbb27e427f74bd, State: Running, Role: LEADER
I20260806 09:16:34.816835 3787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ba8e7312d14926a0fbb27e427f74bd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42227 } }
I20260806 09:16:34.817662 3775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:34.819790 3789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57ba8e7312d14926a0fbb27e427f74bd. Latest consensus state: current_term: 1 leader_uuid: "57ba8e7312d14926a0fbb27e427f74bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ba8e7312d14926a0fbb27e427f74bd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42227 } } }
I20260806 09:16:34.819892 3788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57ba8e7312d14926a0fbb27e427f74bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ba8e7312d14926a0fbb27e427f74bd" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 42227 } } }
I20260806 09:16:34.820250 3789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:34.820343 3788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:34.821260 3798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:34.828073 3798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:34.837045 3798 catalog_manager.cc:1347] Generated new cluster ID: 2c1c43ea81ed46b49cbb7597eddb0ab3
I20260806 09:16:34.837183 3798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:34.882390 3798 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:34.882893 3798 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:34.893417 3798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57ba8e7312d14926a0fbb27e427f74bd: Generated new TSK 0
I20260806 09:16:34.894198 3798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:34.912808 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3675
W20260806 09:16:34.914013 3767 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48263
W20260806 09:16:34.914755 3348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3675: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:16:34.914919 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3675
2026-08-06T09:16:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2265 ms)
[----------] 18 tests from AuthTokenIssuingTest (40849 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:35 dist-test-slave-w4v5 krb5kdc[3818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:35 dist-test-slave-w4v5 krb5kdc[3818](info): set up 2 sockets
Aug 06 09:16:35 dist-test-slave-w4v5 krb5kdc[3818](info): commencing operation
krb5kdc: starting...
W20260806 09:16:36.249624 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:36 dist-test-slave-w4v5 krb5kdc[3818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:16:36.612916 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39423
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:39423,127.3.69.61:39061,127.3.69.60:41817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:36.968293 3838 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:16:36.968698 3838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:36.968832 3838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:36.968979 3838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:36.976344 3838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:36.976466 3838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:36.976574 3838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:36.976673 3838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:36.976765 3838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:36.986795 3838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:39423,127.3.69.61:39061,127.3.69.60:41817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:39423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:36.988238 3838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:36.989609 3838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:37.001721 3843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:37.001808 3844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:37.001748 3847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:37.002702 3838 server_base.cc:1034] running on GCE node
I20260806 09:16:37.003551 3838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:37.005012 3838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:37.006247 3838 hybrid_clock.cc:648] HybridClock initialized: now 1786007797006175 us; error 53 us; skew 500 ppm
I20260806 09:16:37.006573 3838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:37 dist-test-slave-w4v5 krb5kdc[3818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:37.009938 3838 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:16:37.011600 3838 webserver.cc:460] Webserver started at http://127.3.69.62:33211/ using document root <none> and password file <none>
I20260806 09:16:37.012312 3838 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:37.012454 3838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:37.012804 3838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:37.016022 3838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "2b79dd0b582948c0853bee3840143512"
format_stamp: "Formatted at 2026-08-06 09:16:37 on dist-test-slave-w4v5"
server_key: "03b8ac2cbe7c3bc6099e48c372634787"
server_key_iv: "febc12771f0a3effc7d478da29bfec77"
server_key_version: "encryptionkey@0"
I20260806 09:16:37.016865 3838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "2b79dd0b582948c0853bee3840143512"
format_stamp: "Formatted at 2026-08-06 09:16:37 on dist-test-slave-w4v5"
server_key: "03b8ac2cbe7c3bc6099e48c372634787"
server_key_iv: "febc12771f0a3effc7d478da29bfec77"
server_key_version: "encryptionkey@0"
I20260806 09:16:37.022096 3838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:16:37.025628 3860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:37.026940 3838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:16:37.027187 3838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "2b79dd0b582948c0853bee3840143512"
format_stamp: "Formatted at 2026-08-06 09:16:37 on dist-test-slave-w4v5"
server_key: "03b8ac2cbe7c3bc6099e48c372634787"
server_key_iv: "febc12771f0a3effc7d478da29bfec77"
server_key_version: "encryptionkey@0"
I20260806 09:16:37.027428 3838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:37.053846 3838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:37.057971 3838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:37.058283 3838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:37.077893 3838 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:39423
I20260806 09:16:37.077919 3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:39423 every 8 connection(s)
I20260806 09:16:37.079399 3838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:37.083158 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:37.088729 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3838
I20260806 09:16:37.088979 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:37.089677 3348 external_mini_cluster.cc:1428] Setting key 29928606945611ec23b462e958496dad
I20260806 09:16:37.089826 3933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:16:37 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:16:37.115471 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.092895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39423 (local address 127.0.0.1:35070)
0806 09:16:37.093817 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.093869 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:37.096642 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:37.097481 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:37.097490 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:37.098337 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:37.098835 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:37.098863 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.100483 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.100491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:37.102311 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:37.102329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.103334 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.103345 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:37.103566 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:37.104259 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:37.104292 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:37.107142 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:37.111611 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:37.111621 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:37.111630 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:37.112006 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:37.112474 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:37.112477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:37.112479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:37.112650 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:37.113625 (+ 975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:37.113631 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:37.113826 (+ 195us) client_negotiation.cc:769] Sending connection context
0806 09:16:37.114027 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 09:16:37.114395 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":375,"threads_started":1}
I20260806 09:16:37.115567 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.092895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.3.69.62:39423)
0806 09:16:37.094734 (+ 1839us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:37.094739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:37.094782 (+ 43us) server_negotiation.cc:407] Connection header received
0806 09:16:37.097006 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:37.097011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:37.097231 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:37.097386 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:37.099058 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.100312 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.102461 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.103017 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.103230 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:37.107384 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:37.107396 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:37.107413 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:37.107456 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:37.111468 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:37.112147 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:37.112158 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:37.112164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:37.112351 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:37.112769 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:37.112777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:37.112782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:37.113480 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:37.113677 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:37.114219 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 09:16:37.114618 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":1645,"thread_start_us":1410,"threads_started":1}
I20260806 09:16:37.118530 3933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:16:37.119695 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.119274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54084)
0806 09:16:37.119463 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.119588 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:16:37.119968 3864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
I20260806 09:16:37.121594 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.121231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54096)
0806 09:16:37.121384 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.121459 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:16:37.121793 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39061: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.69.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab.
Entry for principal kudu/127.3.69.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab.
I20260806 09:16:37.168577 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.61:39061
--webserver_interface=127.3.69.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab
--principal=kudu/127.3.69.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:39423,127.3.69.61:39061,127.3.69.60:41817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:16:37.171160 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } attempt: 1
I20260806 09:16:37.172479 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.171926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54104)
0806 09:16:37.172173 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.172292 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:16:37.173671 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.173220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54120)
0806 09:16:37.173392 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.173485 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 09:16:37.173871 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39061: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
I20260806 09:16:37.242136 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } attempt: 2
I20260806 09:16:37.243510 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.242908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54126)
0806 09:16:37.243272 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.243365 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 09:16:37.244635 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.244260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54130)
0806 09:16:37.244406 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.244488 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 09:16:37.244820 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39061: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
I20260806 09:16:37.336079 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } attempt: 3
I20260806 09:16:37.337306 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.336837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54138)
0806 09:16:37.337054 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.337158 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 09:16:37.338335 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.337953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54142)
0806 09:16:37.338109 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.338187 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 09:16:37.338534 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39061: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
I20260806 09:16:37.481856 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } attempt: 4
I20260806 09:16:37.483206 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.482589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54146)
0806 09:16:37.482881 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.483006 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260806 09:16:37.484237 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.483887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54158)
0806 09:16:37.484037 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.484124 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 09:16:37.484442 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39061: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39061: connect: Connection refused (error 111)
W20260806 09:16:37.515188 3941 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:16:37.515525 3941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:37.515667 3941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:37.515779 3941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:37.523026 3941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:37.523119 3941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:37.523221 3941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:37.523290 3941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:37.523377 3941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:37.533195 3941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:39423,127.3.69.61:39061,127.3.69.60:41817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab
--principal=kudu/127.3.69.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.61:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.69.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:37.534432 3941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:37.535802 3941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:37.547487 3952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:37.547690 3947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:37.547667 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:37.548512 3941 server_base.cc:1034] running on GCE node
I20260806 09:16:37.549296 3941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:37.550604 3941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:37.551878 3941 hybrid_clock.cc:648] HybridClock initialized: now 1786007797551821 us; error 34 us; skew 500 ppm
I20260806 09:16:37.552238 3941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:37 dist-test-slave-w4v5 krb5kdc[3818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:37.555528 3941 init.cc:377] Logged in from keytab as kudu/127.3.69.61@KRBTEST.COM (short username kudu)
I20260806 09:16:37.556977 3941 webserver.cc:460] Webserver started at http://127.3.69.61:46053/ using document root <none> and password file <none>
I20260806 09:16:37.557607 3941 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:37.557695 3941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:37.557958 3941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:37.561007 3941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/data/instance:
uuid: "8636245ea3764ec2873357ced84d2210"
format_stamp: "Formatted at 2026-08-06 09:16:37 on dist-test-slave-w4v5"
server_key: "568fcea22e4959032d27cb27c98b5b60"
server_key_iv: "8d12f93f48e5dd1cb9a03b756131395c"
server_key_version: "encryptionkey@0"
I20260806 09:16:37.561790 3941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/wal/instance:
uuid: "8636245ea3764ec2873357ced84d2210"
format_stamp: "Formatted at 2026-08-06 09:16:37 on dist-test-slave-w4v5"
server_key: "568fcea22e4959032d27cb27c98b5b60"
server_key_iv: "8d12f93f48e5dd1cb9a03b756131395c"
server_key_version: "encryptionkey@0"
I20260806 09:16:37.566908 3941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 09:16:37.570263 3961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:37.571679 3941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:16:37.571875 3941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/wal
uuid: "8636245ea3764ec2873357ced84d2210"
format_stamp: "Formatted at 2026-08-06 09:16:37 on dist-test-slave-w4v5"
server_key: "568fcea22e4959032d27cb27c98b5b60"
server_key_iv: "8d12f93f48e5dd1cb9a03b756131395c"
server_key_version: "encryptionkey@0"
I20260806 09:16:37.572090 3941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:37.597656 3941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:37.601583 3941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:37.601854 3941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:37.620868 3941 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.61:39061
I20260806 09:16:37.620905 4043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.61:39061 every 8 connection(s)
I20260806 09:16:37.622180 3941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
I20260806 09:16:37.623857 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 3941
I20260806 09:16:37.624078 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-1/wal/instance
I20260806 09:16:37.624519 3348 external_mini_cluster.cc:1428] Setting key 7ca5e48804637329070de10de3a1714a
I20260806 09:16:37.626224 4044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:37.632999 4044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:16:37 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:16:37.655385 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.635525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.3.69.62:39423)
0806 09:16:37.636101 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:37.636109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:37.637016 (+ 907us) server_negotiation.cc:407] Connection header received
0806 09:16:37.638867 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:37.638876 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:37.638978 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:37.639137 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:37.641213 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.642010 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.644476 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.644921 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.645021 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:37.649285 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:37.649290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:37.649293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:37.649338 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:37.652179 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:37.652936 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:37.652944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:37.652947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:37.653092 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:37.653531 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:37.653536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:37.653539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:37.653823 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:37.654019 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:37.654745 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 09:16:37.654967 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":302,"threads_started":1}
I20260806 09:16:37.656338 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.636010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39423 (local address 127.0.0.1:35074)
0806 09:16:37.636831 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.636876 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:37.638605 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:37.639343 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:37.639351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:37.640162 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:37.640988 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:37.641015 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.642179 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.642187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:37.644317 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:37.644335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.645176 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.645187 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:37.645508 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:37.646243 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:37.646277 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:37.649039 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:37.652336 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:37.652346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:37.652362 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:37.652806 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:37.653219 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:37.653224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:37.653228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:37.653410 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:37.653970 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:37.653980 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:37.654364 (+ 384us) client_negotiation.cc:769] Sending connection context
0806 09:16:37.654652 (+ 288us) client_negotiation.cc:240] Negotiation successful
0806 09:16:37.655178 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":338,"threads_started":1}
I20260806 09:16:37.658275 4044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.69.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.60@KRBTEST.COM" created.
Aug 06 09:16:37 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:16:37.674916 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.659185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54160)
0806 09:16:37.659401 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.659426 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:37.659563 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:37.660464 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:37.660468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:37.660878 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:37.661011 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:37.661019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.662732 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.662736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:37.663654 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:37.663666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.664747 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.664759 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:37.664989 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:37.665569 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:37.665597 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:37.668188 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:37.672519 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:37.672523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:37.672525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:37.672897 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:37.673322 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:37.673325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:37.673327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:37.673447 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:37.674177 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:37.674180 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:37.674319 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 09:16:37.674419 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 09:16:37.674599 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 09:16:37.675424 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.659206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.3.69.61:39061)
0806 09:16:37.659914 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:37.659926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:37.659961 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:16:37.660039 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:37.660049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:37.660292 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:37.660449 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:37.661348 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.662573 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.663841 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.664596 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.664700 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:37.668402 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:37.668418 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:37.668436 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:37.668483 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:37.672390 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:37.673041 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:37.673052 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:37.673057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:37.673214 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:37.673559 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:37.673566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:37.673572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:37.674050 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:37.674283 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:37.674629 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 09:16:37.674911 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":303,"threads_started":1}
I20260806 09:16:37.677841 4044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:16:37.678896 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.678475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45796)
0806 09:16:37.678626 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.678735 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 09:16:37.679198 3963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
I20260806 09:16:37.680683 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.680362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45806)
0806 09:16:37.680503 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.680586 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 09:16:37.680877 4044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab.
Entry for principal kudu/127.3.69.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab.
I20260806 09:16:37.704371 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.60:41817
--webserver_interface=127.3.69.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab
--principal=kudu/127.3.69.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:34557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:39423,127.3.69.61:39061,127.3.69.60:41817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:16:37.730176 4044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 1
I20260806 09:16:37.731655 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.730998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45808)
0806 09:16:37.731286 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.731436 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260806 09:16:37.732733 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.732359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45820)
0806 09:16:37.732525 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.732615 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:16:37.732976 4044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
I20260806 09:16:37.783727 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } attempt: 5
Aug 06 09:16:37 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:16:37.798846 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.784514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54162)
0806 09:16:37.784752 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.784770 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:37.784885 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:37.785249 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:37.785253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:37.785645 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:37.785831 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:37.785839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.787276 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.787279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:37.788169 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:37.788182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.789167 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.789172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:37.789300 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:37.789960 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:37.789979 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:37.792195 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:37.796423 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:37.796427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:37.796430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:37.796809 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:37.797250 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:37.797255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:37.797257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:37.797387 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:37.798097 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:37.798101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:37.798238 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 09:16:37.798378 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 09:16:37.798579 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 09:16:37.799050 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.784611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.3.69.61:39061)
0806 09:16:37.784871 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:37.784878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:37.784902 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:16:37.784978 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:37.784983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:37.785088 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:37.785218 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:37.785991 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.787131 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.788331 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:37.789018 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:37.789163 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:37.792396 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:37.792403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:37.792407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:37.792455 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:37.796290 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:37.796933 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:37.796939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:37.796943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:37.797107 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:37.797497 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:37.797502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:37.797505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:37.797979 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:37.798142 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:37.798514 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 09:16:37.798739 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:16:37.800107 3933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:16:37.801074 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.800699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45826)
0806 09:16:37.800862 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.800940 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 09:16:37.801186 4044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 2
I20260806 09:16:37.802136 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.801707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45830)
0806 09:16:37.801884 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.801972 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 09:16:37.802361 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.801960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45844)
0806 09:16:37.802135 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.802233 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 09:16:37.802634 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
I20260806 09:16:37.803331 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.802977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45860)
0806 09:16:37.803161 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.803225 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 09:16:37.803555 4044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
I20260806 09:16:37.853880 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 1
I20260806 09:16:37.855185 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.854582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45872)
0806 09:16:37.854820 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.854939 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 09:16:37.856387 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.855904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45874)
0806 09:16:37.856106 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.856223 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 09:16:37.856657 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
I20260806 09:16:37.894765 4044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 3
I20260806 09:16:37.896085 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.895466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45882)
0806 09:16:37.895727 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.895855 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:16:37.897379 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.896832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45896)
0806 09:16:37.897128 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.897242 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260806 09:16:37.897620 4044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
I20260806 09:16:37.924904 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 2
I20260806 09:16:37.926198 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.925594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45906)
0806 09:16:37.925858 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.925974 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 09:16:37.927393 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:37.926953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45916)
0806 09:16:37.927149 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:37.927253 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 09:16:37.927639 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
I20260806 09:16:38.033895 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 3
I20260806 09:16:38.035308 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.034673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45928)
0806 09:16:38.034931 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.035045 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 09:16:38.036525 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.036005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45930)
0806 09:16:38.036245 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.036368 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260806 09:16:38.036799 3933 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
I20260806 09:16:38.040849 4044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 4
I20260806 09:16:38.042024 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.041489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45936)
0806 09:16:38.041731 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.041829 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 09:16:38.043043 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.042653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45952)
0806 09:16:38.042830 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.042912 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 09:16:38.043437 4044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:41817: Network error: Client connection negotiation failed: client connection to 127.3.69.60:41817: connect: Connection refused (error 111)
W20260806 09:16:38.065454 4056 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:16:38.065830 4056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:38.065960 4056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:38.066092 4056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:38.073557 4056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:38.073649 4056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:38.073751 4056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:38.073818 4056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:38.073904 4056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:38.084138 4056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:34557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:39423,127.3.69.61:39061,127.3.69.60:41817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab
--principal=kudu/127.3.69.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.60:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.69.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:38.085368 4056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:38.086688 4056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:38.099053 4062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:38.099123 4063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:38.099092 4065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:38.100152 4056 server_base.cc:1034] running on GCE node
I20260806 09:16:38.100914 4056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:38.102301 4056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:38.103525 4056 hybrid_clock.cc:648] HybridClock initialized: now 1786007798103463 us; error 35 us; skew 500 ppm
I20260806 09:16:38.103926 4056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:38.107334 4056 init.cc:377] Logged in from keytab as kudu/127.3.69.60@KRBTEST.COM (short username kudu)
I20260806 09:16:38.108753 4056 webserver.cc:460] Webserver started at http://127.3.69.60:36897/ using document root <none> and password file <none>
I20260806 09:16:38.109419 4056 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:38.109504 4056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:38.109766 4056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:38.112759 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/data/instance:
uuid: "e02f51c4db3e4ef891c08a32fbd8308a"
format_stamp: "Formatted at 2026-08-06 09:16:38 on dist-test-slave-w4v5"
server_key: "536c25f39112633d1585fbee6c5d0365"
server_key_iv: "aaf2c391d1fbeecd37aae98c739590ae"
server_key_version: "encryptionkey@0"
I20260806 09:16:38.113492 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/wal/instance:
uuid: "e02f51c4db3e4ef891c08a32fbd8308a"
format_stamp: "Formatted at 2026-08-06 09:16:38 on dist-test-slave-w4v5"
server_key: "536c25f39112633d1585fbee6c5d0365"
server_key_iv: "aaf2c391d1fbeecd37aae98c739590ae"
server_key_version: "encryptionkey@0"
I20260806 09:16:38.118628 4056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:16:38.122090 4075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:38.123562 4056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 09:16:38.123734 4056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/wal
uuid: "e02f51c4db3e4ef891c08a32fbd8308a"
format_stamp: "Formatted at 2026-08-06 09:16:38 on dist-test-slave-w4v5"
server_key: "536c25f39112633d1585fbee6c5d0365"
server_key_iv: "aaf2c391d1fbeecd37aae98c739590ae"
server_key_version: "encryptionkey@0"
I20260806 09:16:38.123960 4056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:38.152112 4056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:38.156612 4056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:38.156929 4056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:38.176872 4056 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.60:41817
I20260806 09:16:38.176903 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.60:41817 every 8 connection(s)
I20260806 09:16:38.178337 4056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
I20260806 09:16:38.178905 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4056
I20260806 09:16:38.179126 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786007418649204-3348-0/minicluster-data/master-2/wal/instance
I20260806 09:16:38.179756 3348 external_mini_cluster.cc:1428] Setting key 79460fd9bb3849173fafd1c44677294f
I20260806 09:16:38.182796 4145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:38.191300 4145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:16:38.204885 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.181753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.3.69.60:41817)
0806 09:16:38.182586 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.182598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.182638 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:16:38.184369 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.184390 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.184743 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.185081 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.187027 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.188445 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.190254 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.190889 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.191023 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.196228 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.196241 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.196255 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.196308 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.200399 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.201242 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.201248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.201254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.201405 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.201869 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.201874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.201877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.202643 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.203003 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.203320 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.203796 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":398,"threads_started":1}
I20260806 09:16:38.212554 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.194411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.3.69.62:39423)
0806 09:16:38.194959 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.194966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.195542 (+ 576us) server_negotiation.cc:407] Connection header received
0806 09:16:38.195733 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.195738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.195840 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.195965 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.197781 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.198669 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.200806 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.201410 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.201552 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.206030 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.206035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.206039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.206082 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.209585 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.210278 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.210286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.210292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.210491 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.210915 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.210920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.210924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.211275 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.211466 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.211933 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.212144 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":249,"threads_started":1}
I20260806 09:16:38.212569 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.194352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:39423 (local address 127.0.0.1:35090)
0806 09:16:38.195391 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.195431 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:38.195584 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:38.196051 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:38.196060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:38.197155 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:38.197557 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.197573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.198863 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.198868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.200661 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.200672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.201603 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.201607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.201747 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.202368 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:38.202389 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:38.204899 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:38.209735 (+ 4836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.209745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.209750 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.210140 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.210622 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.210629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.210633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.210790 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.211412 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:38.211422 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:38.211623 (+ 201us) client_negotiation.cc:769] Sending connection context
0806 09:16:38.211801 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 09:16:38.212127 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":626,"threads_started":1}
I20260806 09:16:38.214008 3933 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 4
I20260806 09:16:38.214553 4145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
I20260806 09:16:38.224959 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.207901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.3.69.62:39423)
0806 09:16:38.208619 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.208626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.208648 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:16:38.208688 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.208691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.208748 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.208837 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.210425 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.211407 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.212635 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.213301 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.213437 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.218774 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.218780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.218783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.218812 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.222149 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.222855 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.222862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.222866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.222993 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.223408 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.223414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.223417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.223668 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.224001 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.224328 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.224556 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"spinlock_wait_cycles":25856,"thread_start_us":349,"threads_started":1}
I20260806 09:16:38.231690 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.214800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45960)
0806 09:16:38.215046 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.215065 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:38.215216 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:38.215719 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:38.215724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:38.216230 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:38.216478 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.216493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.217936 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.217942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.219475 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.219490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.220443 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.220452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.220594 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.221147 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:38.221194 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:38.224283 (+ 3089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:38.228737 (+ 4454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.228742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.228746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.229273 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.229766 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.229772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.229775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.229921 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.230612 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:38.230617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:38.230789 (+ 172us) client_negotiation.cc:769] Sending connection context
0806 09:16:38.230915 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 09:16:38.231260 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260806 09:16:38.231796 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.215472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39061 (local address 127.0.0.1:54170)
0806 09:16:38.215749 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.215768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:38.215857 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:38.216166 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:38.216170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:38.216652 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:38.216879 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.216890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.217998 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.218006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.219436 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.219451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.220618 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.220622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.220749 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.221278 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:38.221299 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:38.224273 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:38.228821 (+ 4548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.228828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.228833 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.229308 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.229788 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.229793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.229797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.229968 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.230730 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:38.230734 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:38.230885 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 09:16:38.231054 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 09:16:38.231432 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":6}
I20260806 09:16:38.232625 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.214844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.3.69.60:41817)
0806 09:16:38.215325 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.215332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.215354 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:16:38.215413 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.215419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.215501 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.216317 (+ 816us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.216632 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.217738 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.219643 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.220284 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.220404 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.224468 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.224476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.224480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.224513 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.228606 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.229422 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.229428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.229430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.229603 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.230045 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.230076 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.230080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.230404 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.231925 (+ 1521us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.232063 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.232288 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":340}
I20260806 09:16:38.232719 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.215689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.3.69.61:39061)
0806 09:16:38.215864 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.215870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.215891 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:16:38.215958 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.215964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.216031 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.216150 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.217058 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.217778 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.219615 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.220282 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.220387 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.224491 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.224498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.224502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.224539 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.228690 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.229452 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.229457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.229461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.229659 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.230116 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.230123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.230127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.230467 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.230666 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.232077 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.232307 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 09:16:38.233722 4145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:16:38.239171 3933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512: Bootstrap starting.
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007798, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
I20260806 09:16:38.242875 3933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:38.244525 3933 log.cc:824] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:38.245402 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.226652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.3.69.61:39061)
0806 09:16:38.227373 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.227381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.227409 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:16:38.227460 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.227465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.227537 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.227680 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.229409 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.230449 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.232052 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.232752 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.232894 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.238871 (+ 5977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.238877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.238881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.238913 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.242370 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.243194 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.243201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.243205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.243342 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.243768 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.243775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.243783 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.244179 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.244454 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.244742 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.244970 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":312,"threads_started":1}
I20260806 09:16:38.248565 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.234354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45968)
0806 09:16:38.234515 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.234533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:38.234631 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:38.234905 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:38.234909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:38.235358 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:38.235488 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.235496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.236464 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.236471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.237634 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.237649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.238382 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.238388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.238594 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.239229 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:38.239256 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:38.241932 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:38.245816 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.245820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.245823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.246293 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.246720 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.246725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.246728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.246906 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.247567 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:38.247571 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:38.247726 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 09:16:38.247891 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 09:16:38.248227 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 09:16:38.249044 3933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512: No bootstrap required, opened a new log
I20260806 09:16:38.249413 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.234422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.3.69.60:41817)
0806 09:16:38.234627 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.234642 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.234659 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:16:38.234728 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.234733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.234794 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.234889 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.235624 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.236340 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.237780 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.238215 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.238342 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.242153 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.242160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.242165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.242200 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.245692 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.246414 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.246419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.246422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.246578 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.247039 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.247047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.247050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.247451 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.247644 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.248844 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.249060 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 09:16:38.250769 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:38.253648 3897 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2b79dd0b582948c0853bee3840143512"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35098
I20260806 09:16:38.255136 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a: Bootstrap starting.
I20260806 09:16:38.259208 4145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:38.260540 4145 log.cc:824] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:38.260959 3933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } }
I20260806 09:16:38.261436 3933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b79dd0b582948c0853bee3840143512, State: Initialized, Role: FOLLOWER
I20260806 09:16:38.261966 3933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } }
I20260806 09:16:38.262957 4164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.263406 4164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:38.263983 4145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a: No bootstrap required, opened a new log
I20260806 09:16:38.264221 3933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:38.264397 3897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 09:16:38.264598 3897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:38.269084 3897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } }
I20260806 09:16:38.272114 3897 leader_election.cc:288] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [CANDIDATE]: Term 1 election: Requested vote from peers 8636245ea3764ec2873357ced84d2210 (127.3.69.61:39061), e02f51c4db3e4ef891c08a32fbd8308a (127.3.69.60:41817)
I20260806 09:16:38.272078 4145 raft_consensus.cc:348] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } }
I20260806 09:16:38.272558 4145 raft_consensus.cc:729] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02f51c4db3e4ef891c08a32fbd8308a, State: Initialized, Role: FOLLOWER
I20260806 09:16:38.273087 4145 consensus_queue.cc:260] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } }
I20260806 09:16:38.275611 4171 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.276505 4145 sys_catalog.cc:564] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:38.276613 4111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b79dd0b582948c0853bee3840143512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e02f51c4db3e4ef891c08a32fbd8308a"
I20260806 09:16:38.277900 4111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:38.282543 4111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b79dd0b582948c0853bee3840143512 in term 1.
I20260806 09:16:38.283420 4171 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:38.283591 3861 leader_election.cc:302] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b79dd0b582948c0853bee3840143512, e02f51c4db3e4ef891c08a32fbd8308a; no voters:
I20260806 09:16:38.284198 4164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:38.291388 4164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [term 1 LEADER]: Becoming Leader. State: Replica: 2b79dd0b582948c0853bee3840143512, State: Running, Role: LEADER
I20260806 09:16:38.292024 4164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } }
I20260806 09:16:38.299561 4186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b79dd0b582948c0853bee3840143512. Latest consensus state: current_term: 1 leader_uuid: "2b79dd0b582948c0853bee3840143512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.300680 4186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: This master's current role is: LEADER
W20260806 09:16:38.311298 3867 tablet.cc:2369] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:16:38.342718 4044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } attempt: 5
I20260806 09:16:38.318841 3838 master_runner.cc:186] Error getting master registration for 127.3.69.61:39061: OK, instance_id { permanent_uuid: "8636245ea3764ec2873357ced84d2210" instance_seqno: 1786007797615479 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:16:38.353229 3838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 09:16:38.362955 4198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:16:38.363124 4198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:16:38.363387 4202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:38.365489 4202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 09:16:38 dist-test-slave-w4v5 krb5kdc[3818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
I20260806 09:16:38.384565 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.343485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:41817 (local address 127.0.0.1:45998)
0806 09:16:38.343663 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:38.343682 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:38.343792 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:38.361402 (+ 17610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:38.361408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:38.361933 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:38.362167 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.362179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.364188 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.364194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.365610 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:38.365626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.366657 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.366664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:38.366875 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.367718 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:38.367754 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:38.371535 (+ 3781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:38.381568 (+ 10033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.381573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.381579 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.382158 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.382626 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:38.382630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:38.382633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:38.382835 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:38.383546 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:38.383551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:38.383717 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 09:16:38.383871 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 09:16:38.384179 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 09:16:38.387267 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.297959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.3.69.62:39423)
0806 09:16:38.298249 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.298257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.298721 (+ 464us) server_negotiation.cc:407] Connection header received
0806 09:16:38.303308 (+ 4587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.303318 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.303428 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.303519 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.336573 (+ 33054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.337735 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.344726 (+ 6991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.346366 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.346489 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.372708 (+ 26219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.372720 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.372729 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.372782 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.379675 (+ 6893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.381417 (+ 1742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.381427 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.381432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.381767 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.382826 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.382863 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.382872 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.383855 (+ 983us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.384482 (+ 627us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.385805 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.386434 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:16:38.389815 4111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [term 1 FOLLOWER]: Refusing update from remote peer 2b79dd0b582948c0853bee3840143512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:16:38.390813 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.299367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.3.69.60:41817)
0806 09:16:38.299587 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.299596 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.335455 (+ 35859us) server_negotiation.cc:407] Connection header received
0806 09:16:38.335643 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.335652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.335747 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.335847 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.345737 (+ 9890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.346887 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.356487 (+ 9600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.357236 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.357344 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.372627 (+ 15283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.372635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.372643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.372697 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.379393 (+ 6696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.385259 (+ 5866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.385264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.385267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.385409 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.385888 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.385894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.385896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.386262 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.386507 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.387114 (+ 607us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.389579 (+ 2465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:16:38.390900 4164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:16:38.391215 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.343626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.3.69.60:41817)
0806 09:16:38.356187 (+ 12561us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.356194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.356224 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:16:38.356283 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.356289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.356378 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.362390 (+ 6012us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.362703 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.363997 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.365814 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.366476 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.367234 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.371776 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.371785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.371791 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.371835 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.381402 (+ 9567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.382331 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.382338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.382342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.382484 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.382991 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.382995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.382998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.383402 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.388344 (+ 4942us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.388470 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.389988 (+ 1518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1294,"server-negotiator.queue_time_us":12310,"thread_start_us":11940,"threads_started":1}
I20260806 09:16:38.394833 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.299442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.3.69.61:39061)
0806 09:16:38.299654 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.299660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.349069 (+ 49409us) server_negotiation.cc:407] Connection header received
0806 09:16:38.349232 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.349237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.349358 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.349463 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.356552 (+ 7089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.358008 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.360161 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.361196 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.361357 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.387598 (+ 26241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:38.387604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:38.387608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:38.387657 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:38.391337 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.392384 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.392389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.392392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.392535 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:38.393049 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:38.393054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:38.393057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:38.393415 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:38.393670 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.394256 (+ 586us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.394482 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:16:38.398794 4044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210: Bootstrap starting.
I20260806 09:16:38.407310 4171 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b79dd0b582948c0853bee3840143512. Latest consensus state: current_term: 1 leader_uuid: "2b79dd0b582948c0853bee3840143512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.407658 4171 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [sys.catalog]: This master's current role is: FOLLOWER
W20260806 09:16:38.409138 4085 tablet.cc:2369] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:16:38.410252 4044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:38.414978 4044 log.cc:824] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:38.416491 4216 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:16:38.418517 4186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b79dd0b582948c0853bee3840143512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.418952 4186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:38.421285 4202 catalog_manager.cc:1347] Generated new cluster ID: 80575625c9b14e75927c3390539f0a7d
I20260806 09:16:38.421402 4202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:38.425268 4171 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b79dd0b582948c0853bee3840143512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.426153 4171 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:38.429057 4044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210: No bootstrap required, opened a new log
I20260806 09:16:38.431629 4056 master_runner.cc:186] Error getting master registration for 127.3.69.61:39061: OK, instance_id { permanent_uuid: "8636245ea3764ec2873357ced84d2210" instance_seqno: 1786007797615479 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:16:38.448581 4056 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:16:38.450771 4202 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:38.453090 4202 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:38.455060 4044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } }
I20260806 09:16:38.455521 4044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8636245ea3764ec2873357ced84d2210, State: Initialized, Role: FOLLOWER
I20260806 09:16:38.456142 4044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } }
I20260806 09:16:38.457453 4224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.458195 4224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:38.458585 4044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:38.458666 4003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b79dd0b582948c0853bee3840143512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8636245ea3764ec2873357ced84d2210"
I20260806 09:16:38.459285 4003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:38.462692 4003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b79dd0b582948c0853bee3840143512 in term 1.
I20260806 09:16:38.466269 4000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [term 1 FOLLOWER]: Refusing update from remote peer 2b79dd0b582948c0853bee3840143512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:16:38.468231 4164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:16:38.476573 4202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512: Generated new TSK 0
I20260806 09:16:38.477811 4202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:38.495503 4224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b79dd0b582948c0853bee3840143512. Latest consensus state: current_term: 1 leader_uuid: "2b79dd0b582948c0853bee3840143512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.495880 4224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:38.496287 4236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b79dd0b582948c0853bee3840143512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.496812 4236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:38.498283 4186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b79dd0b582948c0853bee3840143512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79dd0b582948c0853bee3840143512" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 39423 } } peers { permanent_uuid: "8636245ea3764ec2873357ced84d2210" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39061 } } peers { permanent_uuid: "e02f51c4db3e4ef891c08a32fbd8308a" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 41817 } } }
I20260806 09:16:38.498564 4186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b79dd0b582948c0853bee3840143512 [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:38.510648 4248 catalog_manager.cc:1259] Loaded cluster ID: 80575625c9b14e75927c3390539f0a7d
I20260806 09:16:38.510807 4248 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210: loading cluster ID for follower catalog manager: success
I20260806 09:16:38.514827 4248 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210: acquiring CA information for follower catalog manager: success
I20260806 09:16:38.517545 4248 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8636245ea3764ec2873357ced84d2210: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 09:16:38.585620 4158 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.566228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.3.69.62:39423)
0806 09:16:38.566451 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.566466 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.581573 (+ 15107us) server_negotiation.cc:407] Connection header received
0806 09:16:38.583268 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.583277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.583391 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.583488 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.585208 (+ 1720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35110: BlockingRecv error: recv got EOF from 127.0.0.1:35110 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 09:16:38.588135 4159 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.566943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.3.69.61:39061)
0806 09:16:38.567182 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.567191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.582172 (+ 14981us) server_negotiation.cc:407] Connection header received
0806 09:16:38.582621 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.582628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.582749 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.582838 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.587740 (+ 4902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54198: BlockingRecv error: recv got EOF from 127.0.0.1:54198 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 09:16:38.592015 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.582447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.3.69.60:41817)
0806 09:16:38.582665 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.582671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.589311 (+ 6640us) server_negotiation.cc:407] Connection header received
0806 09:16:38.589844 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.589853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.589963 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.590054 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.591678 (+ 1624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46006: BlockingRecv error: recv got EOF from 127.0.0.1:46006 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:16:38.605671 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.598366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.3.69.61:39061)
0806 09:16:38.598601 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.598611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.599116 (+ 505us) server_negotiation.cc:407] Connection header received
0806 09:16:38.599468 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.599472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.599625 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.599787 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:38.600228 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.600984 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.602876 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.603470 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.603623 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.604116 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:38.604852 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:38.605000 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.605229 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.605313 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 09:16:38.621999 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.616373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.3.69.60:41817)
0806 09:16:38.616632 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.616641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.620290 (+ 3649us) server_negotiation.cc:407] Connection header received
0806 09:16:38.620437 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.620443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.620587 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.620689 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.621754 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46014: BlockingRecv error: recv got EOF from 127.0.0.1:46014 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:16:38.673142 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.664408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.3.69.61:39061)
0806 09:16:38.664663 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.664672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.667680 (+ 3008us) server_negotiation.cc:407] Connection header received
0806 09:16:38.667948 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.667953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.668043 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.668183 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:38.668629 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.669543 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.671197 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.671670 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.671837 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.672252 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:38.672401 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:38.672504 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.672674 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.672757 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 09:16:38.693004 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.688814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.3.69.60:41817)
0806 09:16:38.689089 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.689099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.691657 (+ 2558us) server_negotiation.cc:407] Connection header received
0806 09:16:38.691874 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.691880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.691966 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.692073 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.692820 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46026: BlockingRecv error: recv got EOF from 127.0.0.1:46026 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:16:38.704907 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.698996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.3.69.61:39061)
0806 09:16:38.699220 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.699227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.699960 (+ 733us) server_negotiation.cc:407] Connection header received
0806 09:16:38.700172 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.700178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.700274 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.700383 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:38.700841 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.701623 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.703024 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.703541 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.703685 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.704103 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:38.704230 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:38.704331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.704481 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.704562 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 09:16:38.719177 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.716705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.3.69.60:41817)
0806 09:16:38.716956 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.716965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.717525 (+ 560us) server_negotiation.cc:407] Connection header received
0806 09:16:38.717740 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.717746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.717837 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.717946 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.718898 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46036: BlockingRecv error: recv got EOF from 127.0.0.1:46036 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 09:16:38.732343 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.726181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.3.69.61:39061)
0806 09:16:38.726469 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.726477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.727046 (+ 569us) server_negotiation.cc:407] Connection header received
0806 09:16:38.727296 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.727302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.727393 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.727493 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:38.728037 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.729015 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.730334 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.730868 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.731014 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.731578 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:38.731715 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:38.731819 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.731950 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.732023 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
W20260806 09:16:38.742507 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.740276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.3.69.60:41817)
0806 09:16:38.740629 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.740638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.741179 (+ 541us) server_negotiation.cc:407] Connection header received
0806 09:16:38.741389 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.741395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.741479 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.741586 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.742308 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46048: BlockingRecv error: recv got EOF from 127.0.0.1:46048 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
I20260806 09:16:38.773976 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.767535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.3.69.61:39061)
0806 09:16:38.767742 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.767751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.768162 (+ 411us) server_negotiation.cc:407] Connection header received
0806 09:16:38.768510 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.768517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.768653 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.768729 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:38.769492 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.770536 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.772112 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.772587 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.772726 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.773158 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:38.773262 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:38.773357 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.773519 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.773610 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 09:16:38.807471 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.805173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.3.69.60:41817)
0806 09:16:38.805419 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.805429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.805979 (+ 550us) server_negotiation.cc:407] Connection header received
0806 09:16:38.806218 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.806226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.806311 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.806427 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.807207 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46058: BlockingRecv error: recv got EOF from 127.0.0.1:46058 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:16:38.834017 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.827835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.3.69.61:39061)
0806 09:16:38.828179 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.828186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.828711 (+ 525us) server_negotiation.cc:407] Connection header received
0806 09:16:38.829010 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.829015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.829088 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.829194 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:38.829639 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.830466 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.832236 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.832701 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.832844 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.833269 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:38.833371 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:38.833471 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.833623 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.833701 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
W20260806 09:16:38.842283 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.840058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.3.69.60:41817)
0806 09:16:38.840294 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.840303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.840831 (+ 528us) server_negotiation.cc:407] Connection header received
0806 09:16:38.841050 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.841056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.841163 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.841276 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.842033 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46068: BlockingRecv error: recv got EOF from 127.0.0.1:46068 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:16:38.884124 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.878192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.3.69.61:39061)
0806 09:16:38.878403 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.878412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.879147 (+ 735us) server_negotiation.cc:407] Connection header received
0806 09:16:38.879407 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.879413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.879501 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.879618 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:38.880067 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.880894 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.882232 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.882701 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.882843 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.883322 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:38.883453 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:38.883557 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.883705 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.883787 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 09:16:38.896368 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.890729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.3.69.60:41817)
0806 09:16:38.890943 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.890952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.894887 (+ 3935us) server_negotiation.cc:407] Connection header received
0806 09:16:38.895127 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.895133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.895219 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.895330 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.896094 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46070: BlockingRecv error: recv got EOF from 127.0.0.1:46070 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 09:16:38.970453 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:38.963861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.3.69.61:39061)
0806 09:16:38.964175 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.964185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.964838 (+ 653us) server_negotiation.cc:407] Connection header received
0806 09:16:38.965123 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.965128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.965203 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.965315 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:38.965903 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.966693 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.968187 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:38.968689 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:38.968849 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:38.969250 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:38.969457 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:38.969556 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:38.969911 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 09:16:38.970015 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 09:16:38.985390 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:38.982113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.3.69.60:41817)
0806 09:16:38.982294 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:38.982300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:38.983572 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 09:16:38.983867 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:38.983872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:38.983980 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:38.984053 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:38.985108 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46086: BlockingRecv error: recv got EOF from 127.0.0.1:46086 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:16:39.126274 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:39.120947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.3.69.61:39061)
0806 09:16:39.121231 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.121238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.121518 (+ 280us) server_negotiation.cc:407] Connection header received
0806 09:16:39.121756 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.121761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.121838 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.121947 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:39.122404 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.123190 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.124509 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.124966 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.125111 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:39.125544 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:39.125657 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:39.125761 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:39.125913 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 09:16:39.125993 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260806 09:16:39.134209 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:39.132310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.3.69.60:41817)
0806 09:16:39.132549 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.132555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.132865 (+ 310us) server_negotiation.cc:407] Connection header received
0806 09:16:39.133003 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.133007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.133080 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.133162 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:39.133955 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46094: BlockingRecv error: recv got EOF from 127.0.0.1:46094 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:16:39.399199 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:39.394287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.3.69.61:39061)
0806 09:16:39.394483 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.394495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.394700 (+ 205us) server_negotiation.cc:407] Connection header received
0806 09:16:39.394871 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.394876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.394993 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.395132 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:39.395659 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.396359 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.397646 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.398025 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.398127 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:39.398466 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:39.398563 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:39.398631 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:39.398770 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 09:16:39.398832 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 09:16:39.406744 4204 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:39.405300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.3.69.60:41817)
0806 09:16:39.405496 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.405502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.405521 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:16:39.405594 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.405606 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.405699 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.405810 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:39.406557 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46104: BlockingRecv error: recv got EOF from 127.0.0.1:46104 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:16:39.421315 4216 catalog_manager.cc:1259] Loaded cluster ID: 80575625c9b14e75927c3390539f0a7d
I20260806 09:16:39.421589 4216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a: loading cluster ID for follower catalog manager: success
I20260806 09:16:39.424441 4216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a: acquiring CA information for follower catalog manager: success
I20260806 09:16:39.426281 4216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e02f51c4db3e4ef891c08a32fbd8308a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:16:39.930001 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:39.924520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.3.69.61:39061)
0806 09:16:39.925047 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.925052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.925155 (+ 103us) server_negotiation.cc:407] Connection header received
0806 09:16:39.925322 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.925326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.925398 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.925491 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:39.925964 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.926702 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.928156 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.928588 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.928717 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:39.929252 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:39.929361 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:39.929429 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:39.929606 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 09:16:39.929668 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":275,"threads_started":1}
I20260806 09:16:39.942070 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:39.936310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.3.69.60:41817)
0806 09:16:39.936978 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.936983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.937000 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:16:39.937060 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.937064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.937218 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.937323 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:39.937794 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.938570 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.939917 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.940309 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.940405 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:39.940890 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:39.941420 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:39.941571 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:39.941668 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 09:16:39.941752 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":391,"threads_started":1}
I20260806 09:16:39.959295 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:39.949036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.3.69.61:39061)
0806 09:16:39.949246 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.949255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.952719 (+ 3464us) server_negotiation.cc:407] Connection header received
0806 09:16:39.952998 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.953024 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.953118 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.953224 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:39.954045 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.955216 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.957240 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.957666 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.957803 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:39.958378 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:39.958488 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:39.958569 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:39.958716 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 09:16:39.958801 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:16:39.959803 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:39.949947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.3.69.60:41817)
0806 09:16:39.951693 (+ 1746us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.951701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.953042 (+ 1341us) server_negotiation.cc:407] Connection header received
0806 09:16:39.953211 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.953216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.953291 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.953381 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:39.954172 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.955243 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.957247 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.957626 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.957774 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:39.958299 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:39.958440 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:39.958556 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:39.959422 (+ 866us) server_negotiation.cc:299] Negotiation successful
0806 09:16:39.959498 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1585}
I20260806 09:16:39.959985 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:39.949072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.3.69.62:39423)
0806 09:16:39.951701 (+ 2629us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:39.951710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:39.951734 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:16:39.951866 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:39.951874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:39.952108 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:39.953111 (+ 1003us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:39.953326 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.954367 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.956447 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:39.957104 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:39.957887 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:39.958734 (+ 847us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:39.959050 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:39.959252 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:39.959401 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 09:16:39.959546 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2470,"thread_start_us":2359,"threads_started":1}
I20260806 09:16:39.965495 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3838
I20260806 09:16:39.966814 3924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:40.077860 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3838
I20260806 09:16:40.109220 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 3941
I20260806 09:16:40.110507 4037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:40.220659 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 3941
I20260806 09:16:40.251801 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4056
I20260806 09:16:40.253010 4137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:40.360571 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4056
2026-08-06T09:16:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5450 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:16:40 dist-test-slave-w4v5 krb5kdc[4449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:16:40 dist-test-slave-w4v5 krb5kdc[4449](info): set up 2 sockets
Aug 06 09:16:40 dist-test-slave-w4v5 krb5kdc[4449](info): commencing operation
krb5kdc: starting...
W20260806 09:16:41.699622 3348 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.006s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/test-user.keytab.
Aug 06 09:16:41 dist-test-slave-w4v5 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:16:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.69.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
Entry for principal kudu/127.3.69.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab.
I20260806 09:16:42.048437 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:41969
--webserver_interface=127.3.69.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:41969,127.3.69.61:39859,127.3.69.60:39921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:16:42.394556 4467 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:16:42.394896 4467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:42.395046 4467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:42.395217 4467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:42.402366 4467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:42.402448 4467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:42.402554 4467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:42.402628 4467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:42.402722 4467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:42.412624 4467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:41969,127.3.69.61:39859,127.3.69.60:39921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.62.keytab
--principal=kudu/127.3.69.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.62:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.69.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:42.413923 4467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:42.415262 4467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:42.426791 4475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:42.426875 4480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:42.426868 4474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:42.427937 4467 server_base.cc:1034] running on GCE node
I20260806 09:16:42.428730 4467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:42.430029 4467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:42.431232 4467 hybrid_clock.cc:648] HybridClock initialized: now 1786007802431160 us; error 34 us; skew 500 ppm
I20260806 09:16:42.431582 4467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:42 dist-test-slave-w4v5 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:42.439266 4467 init.cc:377] Logged in from keytab as kudu/127.3.69.62@KRBTEST.COM (short username kudu)
I20260806 09:16:42.440760 4467 webserver.cc:460] Webserver started at http://127.3.69.62:46081/ using document root <none> and password file <none>
I20260806 09:16:42.441403 4467 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:42.441493 4467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:42.441783 4467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:42.444891 4467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/data/instance:
uuid: "5b3a588e1f794c77926a319cb8f39abe"
format_stamp: "Formatted at 2026-08-06 09:16:42 on dist-test-slave-w4v5"
server_key: "c01ae0bc88d4c234c2eea92019e0b419"
server_key_iv: "24bb94ca8dc3488a80542f1e67f4a2dd"
server_key_version: "encryptionkey@0"
I20260806 09:16:42.445609 4467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/wal/instance:
uuid: "5b3a588e1f794c77926a319cb8f39abe"
format_stamp: "Formatted at 2026-08-06 09:16:42 on dist-test-slave-w4v5"
server_key: "c01ae0bc88d4c234c2eea92019e0b419"
server_key_iv: "24bb94ca8dc3488a80542f1e67f4a2dd"
server_key_version: "encryptionkey@0"
I20260806 09:16:42.450598 4467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:16:42.454034 4490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:42.455416 4467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:16:42.455590 4467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/wal
uuid: "5b3a588e1f794c77926a319cb8f39abe"
format_stamp: "Formatted at 2026-08-06 09:16:42 on dist-test-slave-w4v5"
server_key: "c01ae0bc88d4c234c2eea92019e0b419"
server_key_iv: "24bb94ca8dc3488a80542f1e67f4a2dd"
server_key_version: "encryptionkey@0"
I20260806 09:16:42.455801 4467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:42.488520 4467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:42.493054 4467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:42.493362 4467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:42.512871 4467 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.62:41969
I20260806 09:16:42.512894 4565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.62:41969 every 8 connection(s)
I20260806 09:16:42.514269 4467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/data/info.pb
I20260806 09:16:42.517961 4567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:42.523535 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4467
I20260806 09:16:42.523736 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-0/wal/instance
I20260806 09:16:42.524340 3348 external_mini_cluster.cc:1428] Setting key ea30ca96a2fee81ee8c4830a33ca9e33
I20260806 09:16:42.525038 4567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:16:42 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:16:42.552052 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.529045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:41969 (local address 127.0.0.1:48162)
0806 09:16:42.529975 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.530026 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:42.532873 (+ 2847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:42.533766 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:42.533776 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:42.534597 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:42.535140 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:42.535168 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:42.536782 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:42.536791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:42.538689 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:42.538707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:42.539717 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:42.539730 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:42.539962 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:42.540728 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:42.540762 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:42.543468 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:42.547983 (+ 4515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:42.547994 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:42.548003 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:42.548408 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:42.548895 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:42.548898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:42.548900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:42.549046 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:42.550093 (+ 1047us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:42.550098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:42.550318 (+ 220us) client_negotiation.cc:769] Sending connection context
0806 09:16:42.550523 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 09:16:42.550915 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":389,"threads_started":1}
I20260806 09:16:42.552101 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.529045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.3.69.62:41969)
0806 09:16:42.530984 (+ 1939us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:42.530991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:42.531041 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:16:42.533289 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:42.533294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:42.533517 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:42.533676 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:42.535384 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:42.536607 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:42.538845 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:42.539464 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:42.539624 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:42.543711 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:42.543724 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:42.543742 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:42.543786 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:42.547852 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:42.548569 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:42.548580 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:42.548586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:42.548781 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:42.549165 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:42.549173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:42.549179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:42.549944 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:42.550208 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:42.550728 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 09:16:42.551161 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":1759,"thread_start_us":1494,"threads_started":1}
I20260806 09:16:42.554996 4567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:16:42.556216 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.555827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33624)
0806 09:16:42.556005 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.556111 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 09:16:42.556493 4492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
I20260806 09:16:42.558125 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.557773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33632)
0806 09:16:42.557917 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.557999 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:16:42.558353 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39859: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.69.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab.
Entry for principal kudu/127.3.69.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab.
I20260806 09:16:42.603761 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.61:39859
--webserver_interface=127.3.69.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab
--principal=kudu/127.3.69.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:41969,127.3.69.61:39859,127.3.69.60:39921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:16:42.607772 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } attempt: 1
I20260806 09:16:42.608983 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.608538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33646)
0806 09:16:42.608747 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.608836 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 09:16:42.610049 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.609685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33662)
0806 09:16:42.609853 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.609922 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 09:16:42.610235 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39859: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
I20260806 09:16:42.678531 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } attempt: 2
I20260806 09:16:42.679965 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.679312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33674)
0806 09:16:42.679622 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.679753 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 09:16:42.681087 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.680650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33690)
0806 09:16:42.680828 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.680930 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 09:16:42.681327 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39859: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
I20260806 09:16:42.772607 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } attempt: 3
I20260806 09:16:42.774008 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.773391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33692)
0806 09:16:42.773656 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.773791 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 09:16:42.775317 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.774773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33702)
0806 09:16:42.774958 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.775060 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:16:42.775585 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39859: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
I20260806 09:16:42.918876 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } attempt: 4
I20260806 09:16:42.920360 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.919631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33716)
0806 09:16:42.919994 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.920166 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260806 09:16:42.921622 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:42.921148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33724)
0806 09:16:42.921329 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:42.921444 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:16:42.921862 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.61:39859: Network error: Client connection negotiation failed: client connection to 127.3.69.61:39859: connect: Connection refused (error 111)
W20260806 09:16:42.956725 4575 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:16:42.957067 4575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:42.957187 4575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:42.957331 4575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:42.964717 4575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:42.964803 4575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:42.964918 4575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:42.965005 4575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:42.965106 4575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:42.974634 4575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:41969,127.3.69.61:39859,127.3.69.60:39921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.61.keytab
--principal=kudu/127.3.69.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.61:39859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.69.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:42.975895 4575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:42.977226 4575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:42.989533 4588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:42.989565 4590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:42.989609 4585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:42.995775 4575 server_base.cc:1034] running on GCE node
I20260806 09:16:42.996716 4575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:42.998188 4575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:42.999459 4575 hybrid_clock.cc:648] HybridClock initialized: now 1786007802999365 us; error 95 us; skew 500 ppm
I20260806 09:16:42.999815 4575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:42 dist-test-slave-w4v5 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:43.003032 4575 init.cc:377] Logged in from keytab as kudu/127.3.69.61@KRBTEST.COM (short username kudu)
I20260806 09:16:43.004673 4575 webserver.cc:460] Webserver started at http://127.3.69.61:35039/ using document root <none> and password file <none>
I20260806 09:16:43.005441 4575 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:43.005582 4575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:43.005913 4575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:43.009131 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/data/instance:
uuid: "8e01399664af496790e001d561a47b1d"
format_stamp: "Formatted at 2026-08-06 09:16:42 on dist-test-slave-w4v5"
server_key: "d4a75bd111403d63d316d0d7fd0e6d63"
server_key_iv: "1d0b3a05ef2e4ce4c45acb60f135b179"
server_key_version: "encryptionkey@0"
I20260806 09:16:43.009925 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/wal/instance:
uuid: "8e01399664af496790e001d561a47b1d"
format_stamp: "Formatted at 2026-08-06 09:16:42 on dist-test-slave-w4v5"
server_key: "d4a75bd111403d63d316d0d7fd0e6d63"
server_key_iv: "1d0b3a05ef2e4ce4c45acb60f135b179"
server_key_version: "encryptionkey@0"
I20260806 09:16:43.015395 4575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:16:43.018720 4602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:43.020120 4575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:16:43.020315 4575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/wal
uuid: "8e01399664af496790e001d561a47b1d"
format_stamp: "Formatted at 2026-08-06 09:16:42 on dist-test-slave-w4v5"
server_key: "d4a75bd111403d63d316d0d7fd0e6d63"
server_key_iv: "1d0b3a05ef2e4ce4c45acb60f135b179"
server_key_version: "encryptionkey@0"
I20260806 09:16:43.020543 4575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:43.045212 4575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:43.049201 4575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:43.049475 4575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:43.068528 4575 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.61:39859
I20260806 09:16:43.068550 4673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.61:39859 every 8 connection(s)
I20260806 09:16:43.069872 4575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/data/info.pb
I20260806 09:16:43.073575 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:43.076027 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4575
I20260806 09:16:43.076236 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-1/wal/instance
I20260806 09:16:43.076776 3348 external_mini_cluster.cc:1428] Setting key fe8d71fb3b6a1749f93cfafdd7244749
I20260806 09:16:43.082810 4676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:16:43.107779 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.086147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.3.69.62:41969)
0806 09:16:43.086689 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.086698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.087751 (+ 1053us) server_negotiation.cc:407] Connection header received
0806 09:16:43.090624 (+ 2873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.090628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.090702 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.090806 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.092972 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.094136 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.095787 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.096577 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.096726 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.100420 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.100430 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.100437 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.100499 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.104388 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.105114 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.105123 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.105129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.105319 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.105744 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.105752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.105757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.106178 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.106345 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.107008 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.107296 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":265,"threads_started":1}
I20260806 09:16:43.108369 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.086676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:41969 (local address 127.0.0.1:48172)
0806 09:16:43.087539 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.087597 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:43.090323 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:43.091070 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:43.091108 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:43.091930 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:43.092737 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.092763 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.094310 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.094314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.095634 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.095646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.096788 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.096793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.096976 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.097701 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:43.097724 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:43.100161 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:43.104531 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.104537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.104549 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.104953 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.105438 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.105441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.105443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.105620 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.106307 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:43.106312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:43.106616 (+ 304us) client_negotiation.cc:769] Sending connection context
0806 09:16:43.106841 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 09:16:43.107275 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":338,"threads_started":1}
I20260806 09:16:43.110200 4676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.69.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.69.60@KRBTEST.COM" created.
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:16:43.125159 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.111164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33730)
0806 09:16:43.111395 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.111414 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:43.111521 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:43.112537 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:43.112542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:43.112979 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:43.113114 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.113123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.114503 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.114508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.115489 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.115501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.116376 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.116382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.116568 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.117162 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:43.117179 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:43.119563 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:43.122625 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.122631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.122635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.122984 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.123425 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.123429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.123433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.123555 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.124147 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:43.124151 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:43.124308 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 09:16:43.124475 (+ 167us) client_negotiation.cc:240] Negotiation successful
0806 09:16:43.124733 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":1}
I20260806 09:16:43.125195 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.111391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.3.69.61:39859)
0806 09:16:43.112163 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.112169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.112194 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:16:43.112234 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.112239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.112395 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.112546 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.113293 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.114347 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.115625 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.116211 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.116365 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.119773 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.119784 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.119795 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.119829 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.122478 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.123156 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.123163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.123165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.123286 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.123680 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.123691 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.123693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.124007 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.124250 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.124635 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.124835 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":473,"threads_started":1}
I20260806 09:16:43.127885 4676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:16:43.129019 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.128557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58774)
0806 09:16:43.128745 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.128860 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 09:16:43.129284 4609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
I20260806 09:16:43.130831 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.130497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58786)
0806 09:16:43.130647 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.130728 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:16:43.131106 4676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.69.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab.
Entry for principal kudu/127.3.69.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab.
I20260806 09:16:43.156507 3348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6yYR2E/build/asan/bin/kudu
/tmp/dist-test-task6yYR2E/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.60:39921
--webserver_interface=127.3.69.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab
--principal=kudu/127.3.69.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.69.20:37569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.69.62:41969,127.3.69.61:39859,127.3.69.60:39921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:16:43.180465 4676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 1
I20260806 09:16:43.181838 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.181210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58790)
0806 09:16:43.181487 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.181615 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 09:16:43.183187 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.182603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58796)
0806 09:16:43.182843 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.182954 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260806 09:16:43.183485 4676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
I20260806 09:16:43.221119 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } attempt: 5
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:16:43.234952 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.222018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.3.69.61:39859)
0806 09:16:43.222197 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.222205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.222229 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:16:43.222475 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.222479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.222630 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.222756 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.223688 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.224435 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.225735 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.226193 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.226364 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.229542 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.229547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.229550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.229588 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.232245 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.232982 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.232992 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.232994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.233110 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.233497 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.233502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.233512 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.233939 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.234097 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.234558 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.234736 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:16:43.235428 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.221911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33742)
0806 09:16:43.222145 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.222166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:43.222312 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:43.222792 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:43.222797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:43.223299 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:43.223527 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.223542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.224584 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.224591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.225489 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.225502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.226388 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.226394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.226608 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.227285 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:43.227303 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:43.229346 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:43.232376 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.232382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.232387 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.232855 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.233232 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.233237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.233241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.233378 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.234092 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:43.234098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:43.234283 (+ 185us) client_negotiation.cc:769] Sending connection context
0806 09:16:43.234455 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 09:16:43.234822 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 09:16:43.236408 4567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:16:43.237426 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.237061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58802)
0806 09:16:43.237224 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.237297 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 09:16:43.238333 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.238030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58818)
0806 09:16:43.238167 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.238232 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:16:43.238618 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
I20260806 09:16:43.251703 4676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 2
I20260806 09:16:43.252789 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.252222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58832)
0806 09:16:43.252457 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.252580 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 09:16:43.253953 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.253564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58844)
0806 09:16:43.253723 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.253810 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 09:16:43.254180 4676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
I20260806 09:16:43.289851 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 1
I20260806 09:16:43.291252 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.290695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58860)
0806 09:16:43.290906 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.291000 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:16:43.292332 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.292006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58874)
0806 09:16:43.292153 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.292224 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 09:16:43.292598 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
I20260806 09:16:43.345453 4676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 3
I20260806 09:16:43.346817 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.346219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58884)
0806 09:16:43.346461 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.346582 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 09:16:43.348182 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.347694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58900)
0806 09:16:43.347936 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.348031 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260806 09:16:43.348455 4676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
I20260806 09:16:43.360811 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 2
I20260806 09:16:43.362016 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.361572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58912)
0806 09:16:43.361772 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.361860 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 09:16:43.363237 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.362844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58928)
0806 09:16:43.362990 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.363059 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 09:16:43.363540 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
I20260806 09:16:43.469769 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 3
I20260806 09:16:43.471012 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.470525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58930)
0806 09:16:43.470758 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.470846 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 09:16:43.472164 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.471774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58938)
0806 09:16:43.471976 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.472057 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260806 09:16:43.472437 4567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
I20260806 09:16:43.491672 4676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 4
I20260806 09:16:43.492851 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.492287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58940)
0806 09:16:43.492533 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.492639 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 09:16:43.493883 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.493492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58948)
0806 09:16:43.493652 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.493747 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 09:16:43.494093 4676 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.69.60:39921: Network error: Client connection negotiation failed: client connection to 127.3.69.60:39921: connect: Connection refused (error 111)
W20260806 09:16:43.507304 4686 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:16:43.507637 4686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:16:43.507737 4686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:16:43.507867 4686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:16:43.515483 4686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:16:43.515592 4686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:16:43.515694 4686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:16:43.515798 4686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:16:43.515890 4686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:16:43.526224 4686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.69.20:37569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.69.62:41969,127.3.69.61:39859,127.3.69.60:39921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/krb5kdc/kudu_127.3.69.60.keytab
--principal=kudu/127.3.69.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.69.60:39921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.69.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 08:04:30 UTC on ccycloud.quasar-tkivct.root.comops.site
build id 4569871
ASAN enabled
I20260806 09:16:43.527565 4686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:16:43.528841 4686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:16:43.540424 4695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:43.540541 4693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:16:43.540566 4692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:16:43.541617 4686 server_base.cc:1034] running on GCE node
I20260806 09:16:43.542445 4686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:16:43.543848 4686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:16:43.545105 4686 hybrid_clock.cc:648] HybridClock initialized: now 1786007803545026 us; error 50 us; skew 500 ppm
I20260806 09:16:43.545460 4686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:16:43.549042 4686 init.cc:377] Logged in from keytab as kudu/127.3.69.60@KRBTEST.COM (short username kudu)
I20260806 09:16:43.550616 4686 webserver.cc:460] Webserver started at http://127.3.69.60:46669/ using document root <none> and password file <none>
I20260806 09:16:43.551437 4686 fs_manager.cc:359] Metadata directory not provided
I20260806 09:16:43.551573 4686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:16:43.551923 4686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:16:43.554939 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/data/instance:
uuid: "de027cec18974d9692fd87dd0577d55e"
format_stamp: "Formatted at 2026-08-06 09:16:43 on dist-test-slave-w4v5"
server_key: "5e6b4518bad64bc153e940cc3ae2692c"
server_key_iv: "0c9278f71ef879ebaefdcd4c93a3914f"
server_key_version: "encryptionkey@0"
I20260806 09:16:43.555842 4686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/wal/instance:
uuid: "de027cec18974d9692fd87dd0577d55e"
format_stamp: "Formatted at 2026-08-06 09:16:43 on dist-test-slave-w4v5"
server_key: "5e6b4518bad64bc153e940cc3ae2692c"
server_key_iv: "0c9278f71ef879ebaefdcd4c93a3914f"
server_key_version: "encryptionkey@0"
I20260806 09:16:43.561318 4686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:16:43.564941 4708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:16:43.566316 4686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:16:43.566521 4686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/data,/tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/wal
uuid: "de027cec18974d9692fd87dd0577d55e"
format_stamp: "Formatted at 2026-08-06 09:16:43 on dist-test-slave-w4v5"
server_key: "5e6b4518bad64bc153e940cc3ae2692c"
server_key_iv: "0c9278f71ef879ebaefdcd4c93a3914f"
server_key_version: "encryptionkey@0"
I20260806 09:16:43.566761 4686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:16:43.585815 4686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:16:43.590507 4686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:16:43.590812 4686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:16:43.610283 4686 rpc_server.cc:307] RPC server started. Bound to: 127.3.69.60:39921
I20260806 09:16:43.610316 4780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.69.60:39921 every 8 connection(s)
I20260806 09:16:43.611764 4686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/data/info.pb
I20260806 09:16:43.615587 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:16:43.618955 3348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6yYR2E/build/asan/bin/kudu as pid 4686
I20260806 09:16:43.619185 3348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6yYR2E/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786007418649204-3348-0/minicluster-data/master-2/wal/instance
I20260806 09:16:43.619647 3348 external_mini_cluster.cc:1428] Setting key 74416f3290fc61eb79c36ae610c84306
I20260806 09:16:43.622732 4781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
I20260806 09:16:43.643688 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.626951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.3.69.62:41969)
0806 09:16:43.628340 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.628347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.628367 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:16:43.628425 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.628430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.628515 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.628622 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.630099 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.631071 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.632981 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.633385 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.633484 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.637499 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.637505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.637508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.637543 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.640532 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.641314 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.641321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.641324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.641475 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.641929 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.641936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.641939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.642273 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.642468 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.642942 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.643212 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269,"thread_start_us":1105,"threads_started":1}
I20260806 09:16:43.644244 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.621336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.3.69.60:39921)
0806 09:16:43.622084 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.622095 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.622135 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:16:43.624046 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.624068 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.624465 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.624840 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.626662 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.628777 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.630605 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.631566 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.631705 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.636749 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.636757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.636769 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.636804 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.639977 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.640676 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.640690 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.640696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.640855 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.641248 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.641253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.641256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.641791 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.642120 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.642581 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.643044 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":344,"threads_started":1}
I20260806 09:16:43.644297 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.626893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.62:41969 (local address 127.0.0.1:48184)
0806 09:16:43.627790 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.627853 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:43.628065 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:43.628696 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:43.628701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:43.629554 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:43.629912 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.629925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.631276 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.631280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.632818 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.632836 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.633649 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.633658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.633893 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.634546 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:43.634590 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:43.637284 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:43.640687 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.640701 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.640709 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.641173 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.641619 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.641625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.641629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.641803 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.642424 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:43.642432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:43.642631 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 09:16:43.642829 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 09:16:43.643209 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":347,"threads_started":1}
I20260806 09:16:43.646814 4781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:16:43.649673 4567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 4
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.62@KRBTEST.COM
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.62@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
I20260806 09:16:43.663259 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.647805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.3.69.61:39859)
0806 09:16:43.647991 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.647996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.648027 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:16:43.648122 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.648127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.648201 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.648306 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.648992 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.649668 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.651452 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.651875 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.652025 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.656015 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.656022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.656027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.656069 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.659627 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.660541 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.660547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.660551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.660784 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.661380 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.661387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.661392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.661791 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.661950 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.662585 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.662828 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 09:16:43.663360 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.647708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.61:39859 (local address 127.0.0.1:33750)
0806 09:16:43.647890 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.647909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:43.648003 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:43.648326 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:43.648331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:43.648710 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:43.648837 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.648845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.649827 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.649833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.651272 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.651283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.652023 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.652029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.652269 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.652891 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:43.652916 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:43.655801 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:43.659774 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.659780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.659784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.660364 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.660991 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.660998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.661008 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.661224 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.662074 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:43.662080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:43.662269 (+ 189us) client_negotiation.cc:769] Sending connection context
0806 09:16:43.662456 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 09:16:43.662882 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":31}
I20260806 09:16:43.664602 4781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:16:43.666944 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.647582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48192 (local address 127.3.69.62:41969)
0806 09:16:43.648104 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.648109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.648122 (+ 13us) server_negotiation.cc:407] Connection header received
0806 09:16:43.648169 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.648173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.648237 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.648365 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.649599 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.650710 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.652622 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.653137 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.653284 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.658510 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.658518 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.658524 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.658568 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.662396 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.663389 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.663395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.663399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.663567 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.664096 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.664101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.664104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.664508 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.664854 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.666345 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.666574 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420}
I20260806 09:16:43.667332 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.650326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58954)
0806 09:16:43.650583 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.650602 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:43.650695 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:43.651194 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:43.651200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:43.651749 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:43.651976 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.651988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.653624 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.653632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.655063 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.655115 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.656188 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.656195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.656404 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.656987 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:43.657013 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:43.660357 (+ 3344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:43.664394 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.664400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.664404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.664960 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.665425 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.665433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.665439 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.665618 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.666362 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:43.666368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:43.666530 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 09:16:43.666659 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 09:16:43.666870 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260806 09:16:43.667937 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.650378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.3.69.60:39921)
0806 09:16:43.650878 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.650883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.650898 (+ 15us) server_negotiation.cc:407] Connection header received
0806 09:16:43.650937 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.650942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.651021 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.651148 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.652174 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.653453 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.655259 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.656025 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.656172 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.660652 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.660661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.660666 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.660706 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.664278 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.665112 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.665117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.665120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.665311 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.665731 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.665743 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.665747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.666208 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.666386 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.666801 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.667054 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":395}
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.60@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
I20260806 09:16:43.675029 4567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe: Bootstrap starting.
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.69.61@KRBTEST.COM
I20260806 09:16:43.679812 4567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:43.681360 4567 log.cc:824] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:43.682730 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.665541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58966)
0806 09:16:43.667695 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.667717 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:43.667844 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:43.668445 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:43.668450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:43.668940 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:43.669154 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.669165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.670454 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.670461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.671591 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.671603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.672268 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.672273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.672522 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.673062 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:43.673083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:43.675204 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:43.679484 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.679490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.679493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.680146 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.680557 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.680564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.680570 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.680710 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.681427 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:43.681431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:43.681594 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 09:16:43.681743 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 09:16:43.682082 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2026,"mutex_wait_us":19}
I20260806 09:16:43.682751 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.665819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.3.69.60:39921)
0806 09:16:43.666419 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.666426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.667799 (+ 1373us) server_negotiation.cc:407] Connection header received
0806 09:16:43.667976 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.667981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.668064 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.668173 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.669326 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.670349 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.671718 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.672129 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.672261 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.675441 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.675452 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.675458 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.675506 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.679331 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.680282 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.680287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.680291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.680433 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.680830 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.680835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.680838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.681296 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.681457 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.681838 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.682224 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":212,"server-negotiator.queue_time_us":449,"thread_start_us":294,"threads_started":1}
I20260806 09:16:43.688087 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.668818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.3.69.61:39859)
0806 09:16:43.668995 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.668999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.669239 (+ 240us) server_negotiation.cc:407] Connection header received
0806 09:16:43.669463 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.669471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.669558 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.669654 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.671625 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.672800 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.674703 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.675387 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.675528 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.680164 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.680173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.680180 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.680233 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.684736 (+ 4503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.685627 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.685632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.685636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.685794 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.686313 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.686319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.686323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.686734 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.687032 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.687432 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.687713 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 09:16:43.688331 4781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: Bootstrap starting.
I20260806 09:16:43.688659 4567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe: No bootstrap required, opened a new log
I20260806 09:16:43.692493 4781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:43.693923 4781 log.cc:824] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:43.693940 3348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:16:43.696935 4781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: No bootstrap required, opened a new log
I20260806 09:16:43.697664 4532 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5b3a588e1f794c77926a319cb8f39abe"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48192
I20260806 09:16:43.699499 4567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } }
I20260806 09:16:43.699862 4567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3a588e1f794c77926a319cb8f39abe, State: Initialized, Role: FOLLOWER
I20260806 09:16:43.700350 4567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } }
I20260806 09:16:43.701618 4797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.702221 4797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:43.702550 4567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:43.702648 4532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 09:16:43.702849 4532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:43.706882 4532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } }
I20260806 09:16:43.708264 4781 raft_consensus.cc:348] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } }
I20260806 09:16:43.708675 4532 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [CANDIDATE]: Term 1 election: Requested vote from peers 8e01399664af496790e001d561a47b1d (127.3.69.61:39859), de027cec18974d9692fd87dd0577d55e (127.3.69.60:39921)
I20260806 09:16:43.708731 4781 raft_consensus.cc:729] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de027cec18974d9692fd87dd0577d55e, State: Initialized, Role: FOLLOWER
I20260806 09:16:43.709260 4781 consensus_queue.cc:260] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } }
I20260806 09:16:43.710719 4809 sys_catalog.cc:455] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.711434 4809 sys_catalog.cc:458] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:43.712008 4748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b3a588e1f794c77926a319cb8f39abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "de027cec18974d9692fd87dd0577d55e"
I20260806 09:16:43.712494 4748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:43.715607 4748 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b3a588e1f794c77926a319cb8f39abe in term 1.
I20260806 09:16:43.711983 4781 sys_catalog.cc:564] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:43.716513 4496 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b3a588e1f794c77926a319cb8f39abe, de027cec18974d9692fd87dd0577d55e; no voters:
I20260806 09:16:43.717279 4797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:16:43.731336 4797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [term 1 LEADER]: Becoming Leader. State: Replica: 5b3a588e1f794c77926a319cb8f39abe, State: Running, Role: LEADER
I20260806 09:16:43.732064 4797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } }
W20260806 09:16:43.735620 4816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:16:43.735769 4816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 09:16:43.746251 4499 tablet.cc:2369] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:16:43.749656 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.715314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.3.69.62:41969)
0806 09:16:43.715545 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.715552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.718476 (+ 2924us) server_negotiation.cc:407] Connection header received
0806 09:16:43.719588 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.719593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.719686 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.719770 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.726130 (+ 6360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.727611 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.732696 (+ 5085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.733337 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.733435 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.741160 (+ 7725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.741167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.741172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.741228 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.746123 (+ 4895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.746944 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.746950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.746954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.747162 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.747656 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.747662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.747665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.748046 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.748323 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.748984 (+ 661us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.749237 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:16:43.760327 4821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3a588e1f794c77926a319cb8f39abe. Latest consensus state: current_term: 1 leader_uuid: "5b3a588e1f794c77926a319cb8f39abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.760855 4821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:43.775982 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.717227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.3.69.61:39859)
0806 09:16:43.717433 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.717440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.728506 (+ 11066us) server_negotiation.cc:407] Connection header received
0806 09:16:43.729830 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.729838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.729933 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.730031 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.738318 (+ 8287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.739605 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.754794 (+ 15189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.755891 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.756037 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.766403 (+ 10366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.766413 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.766420 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.766465 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.770347 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.772227 (+ 1880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.772237 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.772243 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.772458 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.773495 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.773505 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.773511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.774021 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.774261 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.775293 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.775573 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:16:43.782253 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.720167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.3.69.60:39921)
0806 09:16:43.720339 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.720345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.730182 (+ 9837us) server_negotiation.cc:407] Connection header received
0806 09:16:43.730664 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.730670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.730764 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.730852 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.735493 (+ 4641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.736741 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.741130 (+ 4389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.742134 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.742270 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.758512 (+ 16242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.758524 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.758531 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.758583 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.762878 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.765148 (+ 2270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.765184 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.765188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.765471 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.766392 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.766398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.766402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.767207 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.767697 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.781415 (+ 13718us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.781707 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 09:16:43.785406 4837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:16:43.764276 4467 master_runner.cc:186] Error getting master registration for 127.3.69.61:39859: OK, instance_id { permanent_uuid: "8e01399664af496790e001d561a47b1d" instance_seqno: 1786007803063265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:16:43.788877 4467 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:16:43.790057 4837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:16:43.793455 4676 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } attempt: 5
I20260806 09:16:43.800567 4748 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [term 1 FOLLOWER]: Refusing update from remote peer 5b3a588e1f794c77926a319cb8f39abe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:16:43.801537 4821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [LEADER]: Connected to new peer: Peer: permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 06 09:16:43 dist-test-slave-w4v5 krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786007802, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.69.61@KRBTEST.COM for kudu/127.3.69.60@KRBTEST.COM
I20260806 09:16:43.825357 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.796195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.3.69.60:39921)
0806 09:16:43.796397 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:43.796404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:43.796424 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:16:43.796482 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:43.796489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:43.796567 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:43.796681 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:43.797994 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.799251 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.801665 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.802656 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.802818 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.813800 (+ 10982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:16:43.813807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:16:43.813813 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:16:43.813883 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:16:43.818258 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.820087 (+ 1829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.820093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.820096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.820268 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:16:43.821400 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:16:43.821410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:16:43.821416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:16:43.822017 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:16:43.822241 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:43.823814 (+ 1573us) server_negotiation.cc:299] Negotiation successful
0806 09:16:43.824435 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:16:43.827739 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:43.795054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.69.60:39921 (local address 127.0.0.1:58996)
0806 09:16:43.795721 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 09:16:43.795745 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:16:43.795879 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:16:43.796751 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:16:43.796760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:16:43.797478 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:16:43.797800 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.797818 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.799396 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.799406 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.801433 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:16:43.801454 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:16:43.803131 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:16:43.803140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:16:43.803432 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:43.806848 (+ 3416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:16:43.806906 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:16:43.813441 (+ 6535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:16:43.818600 (+ 5159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.818606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.818612 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.819843 (+ 1231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.820623 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:16:43.820632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:16:43.820674 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:16:43.821164 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:16:43.822464 (+ 1300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:16:43.822472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:16:43.822980 (+ 508us) client_negotiation.cc:769] Sending connection context
0806 09:16:43.826101 (+ 3121us) client_negotiation.cc:240] Negotiation successful
0806 09:16:43.826865 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485}
I20260806 09:16:43.834263 4676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d: Bootstrap starting.
I20260806 09:16:43.838223 4676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d: Neither blocks nor log segments found. Creating new log.
I20260806 09:16:43.839725 4676 log.cc:824] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:16:43.841127 4809 sys_catalog.cc:455] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3a588e1f794c77926a319cb8f39abe. Latest consensus state: current_term: 1 leader_uuid: "5b3a588e1f794c77926a319cb8f39abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.841535 4809 sys_catalog.cc:458] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [sys.catalog]: This master's current role is: FOLLOWER
W20260806 09:16:43.844463 4715 tablet.cc:2369] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:16:43.847998 4821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b3a588e1f794c77926a319cb8f39abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.848340 4821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:43.849639 4849 mvcc.cc:205] Tried to move back new op lower bound from 7315487964354957312 to 7315487964094377984. Current Snapshot: MvccSnapshot[applied={T|T < 7315487964354957312}]
I20260806 09:16:43.850039 4837 catalog_manager.cc:1347] Generated new cluster ID: b06847cc75544a50a47ab3fa8b00d276
I20260806 09:16:43.850138 4837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:16:43.851202 4809 sys_catalog.cc:455] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b3a588e1f794c77926a319cb8f39abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.851552 4809 sys_catalog.cc:458] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:43.852069 4846 mvcc.cc:205] Tried to move back new op lower bound from 7315487964354957312 to 7315487964094377984. Current Snapshot: MvccSnapshot[applied={T|T < 7315487964354957312 or (T in {7315487964354957312})}]
I20260806 09:16:43.869887 4676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d: No bootstrap required, opened a new log
I20260806 09:16:43.886255 4853 catalog_manager.cc:1259] Loaded cluster ID: b06847cc75544a50a47ab3fa8b00d276
I20260806 09:16:43.886368 4853 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: loading cluster ID for follower catalog manager: success
I20260806 09:16:43.873728 4686 master_runner.cc:186] Error getting master registration for 127.3.69.61:39859: OK, instance_id { permanent_uuid: "8e01399664af496790e001d561a47b1d" instance_seqno: 1786007803063265 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:16:43.886711 4686 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 09:16:43.887636 4853 catalog_manager.cc:1576] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260806 09:16:43.887776 4853 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:16:43.889222 4676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } }
I20260806 09:16:43.890918 4676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e01399664af496790e001d561a47b1d, State: Initialized, Role: FOLLOWER
I20260806 09:16:43.891597 4676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } }
I20260806 09:16:43.902319 4837 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:16:43.903656 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.904114 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:43.904178 4837 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:16:43.905160 4676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:16:43.905243 4637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b3a588e1f794c77926a319cb8f39abe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e01399664af496790e001d561a47b1d"
I20260806 09:16:43.905763 4637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:16:43.914141 4637 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b3a588e1f794c77926a319cb8f39abe in term 1.
I20260806 09:16:43.915061 4635 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [term 1 FOLLOWER]: Refusing update from remote peer 5b3a588e1f794c77926a319cb8f39abe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:16:43.917490 4821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:16:43.923408 4837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe: Generated new TSK 0
I20260806 09:16:43.924744 4837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:16:43.956709 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3a588e1f794c77926a319cb8f39abe. Latest consensus state: current_term: 1 leader_uuid: "5b3a588e1f794c77926a319cb8f39abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.957105 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:43.957576 4870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b3a588e1f794c77926a319cb8f39abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.957994 4870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:16:43.960493 4821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b3a588e1f794c77926a319cb8f39abe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a588e1f794c77926a319cb8f39abe" member_type: VOTER last_known_addr { host: "127.3.69.62" port: 41969 } } peers { permanent_uuid: "8e01399664af496790e001d561a47b1d" member_type: VOTER last_known_addr { host: "127.3.69.61" port: 39859 } } peers { permanent_uuid: "de027cec18974d9692fd87dd0577d55e" member_type: VOTER last_known_addr { host: "127.3.69.60" port: 39921 } } }
I20260806 09:16:43.960777 4821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3a588e1f794c77926a319cb8f39abe [sys.catalog]: This master's current role is: LEADER
I20260806 09:16:44.016813 4889 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260806 09:16:44.050925 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.037252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.3.69.61:39859)
0806 09:16:44.037428 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.037434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.048617 (+ 11183us) server_negotiation.cc:407] Connection header received
0806 09:16:44.049126 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.049133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.049234 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.049313 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.050667 (+ 1354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33780: BlockingRecv error: recv got EOF from 127.0.0.1:33780 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 09:16:44.052786 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.043357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.3.69.60:39921)
0806 09:16:44.043542 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.043548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.050609 (+ 7061us) server_negotiation.cc:407] Connection header received
0806 09:16:44.051064 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.051087 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.051182 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.051263 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.052516 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59010: BlockingRecv error: recv got EOF from 127.0.0.1:59010 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 09:16:44.054943 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.037155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.3.69.62:41969)
0806 09:16:44.037362 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.037369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.043679 (+ 6310us) server_negotiation.cc:407] Connection header received
0806 09:16:44.053477 (+ 9798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.053484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.053571 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.053679 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.054710 (+ 1031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48216: BlockingRecv error: recv got EOF from 127.0.0.1:48216 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 09:16:44.061264 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.059001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.3.69.61:39859)
0806 09:16:44.059193 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.059199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.059700 (+ 501us) server_negotiation.cc:407] Connection header received
0806 09:16:44.059931 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.059937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.060081 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.060189 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.060995 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33792: BlockingRecv error: recv got EOF from 127.0.0.1:33792 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 09:16:44.068027 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.066000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.3.69.61:39859)
0806 09:16:44.066165 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.066172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.066644 (+ 472us) server_negotiation.cc:407] Connection header received
0806 09:16:44.066828 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.066833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.066922 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.067028 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.067819 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33794: BlockingRecv error: recv got EOF from 127.0.0.1:33794 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 09:16:44.104153 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.101569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.3.69.61:39859)
0806 09:16:44.101768 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.101774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.102301 (+ 527us) server_negotiation.cc:407] Connection header received
0806 09:16:44.102502 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.102509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.102619 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.102755 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.103904 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33810: BlockingRecv error: recv got EOF from 127.0.0.1:33810 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 09:16:44.114302 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.112211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.3.69.61:39859)
0806 09:16:44.112377 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.112383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.112880 (+ 497us) server_negotiation.cc:407] Connection header received
0806 09:16:44.113057 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.113063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.113158 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.113270 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.114088 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33818: BlockingRecv error: recv got EOF from 127.0.0.1:33818 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 09:16:44.128562 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.126165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.3.69.61:39859)
0806 09:16:44.126384 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.126390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.126937 (+ 547us) server_negotiation.cc:407] Connection header received
0806 09:16:44.127175 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.127181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.127291 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.127410 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.128292 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33822: BlockingRecv error: recv got EOF from 127.0.0.1:33822 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 09:16:44.150866 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.148480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.3.69.61:39859)
0806 09:16:44.148674 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.148680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.149295 (+ 615us) server_negotiation.cc:407] Connection header received
0806 09:16:44.149484 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.149497 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.149603 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.149728 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.150611 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33832: BlockingRecv error: recv got EOF from 127.0.0.1:33832 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 09:16:44.190936 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.188177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.3.69.61:39859)
0806 09:16:44.188780 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.188787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.189315 (+ 528us) server_negotiation.cc:407] Connection header received
0806 09:16:44.189528 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.189533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.189626 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.189738 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.190675 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33836: BlockingRecv error: recv got EOF from 127.0.0.1:33836 (error 108)
Metrics: {"server-negotiator.queue_time_us":440}
W20260806 09:16:44.263706 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.261136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.3.69.61:39859)
0806 09:16:44.261366 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.261374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.261993 (+ 619us) server_negotiation.cc:407] Connection header received
0806 09:16:44.262233 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.262240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.262349 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.262478 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.263451 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33838: BlockingRecv error: recv got EOF from 127.0.0.1:33838 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 09:16:44.398335 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.396499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.3.69.61:39859)
0806 09:16:44.396784 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.396792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.396961 (+ 169us) server_negotiation.cc:407] Connection header received
0806 09:16:44.397122 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.397127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.397203 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.397291 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.398024 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33854: BlockingRecv error: recv got EOF from 127.0.0.1:33854 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
W20260806 09:16:44.661729 4683 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:16:44.659744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.3.69.61:39859)
0806 09:16:44.659976 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:44.659982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:44.660232 (+ 250us) server_negotiation.cc:407] Connection header received
0806 09:16:44.660497 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:44.660510 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:44.660587 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:44.660704 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:16:44.661544 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33868: BlockingRecv error: recv got EOF from 127.0.0.1:33868 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:16:44.891220 4853 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: acquiring CA information for follower catalog manager: success
I20260806 09:16:44.893096 4853 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de027cec18974d9692fd87dd0577d55e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:16:45.021481 4889 catalog_manager.cc:1259] Loaded cluster ID: b06847cc75544a50a47ab3fa8b00d276
I20260806 09:16:45.021649 4889 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d: loading cluster ID for follower catalog manager: success
I20260806 09:16:45.024470 4889 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d: acquiring CA information for follower catalog manager: success
I20260806 09:16:45.026326 4889 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e01399664af496790e001d561a47b1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:16:45.193312 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:45.185279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.3.69.61:39859)
0806 09:16:45.186078 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:45.186085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:45.186384 (+ 299us) server_negotiation.cc:407] Connection header received
0806 09:16:45.186604 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:45.186608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:45.186774 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:45.186977 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:45.187563 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.188644 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.190682 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.191224 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.191318 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:45.191983 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:45.192614 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:45.192745 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:45.192889 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 09:16:45.192958 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":414,"threads_started":1}
I20260806 09:16:45.207908 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:45.201251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.3.69.60:39921)
0806 09:16:45.201961 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:45.201965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:45.201982 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:16:45.202078 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:45.202082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:45.202246 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:45.202392 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:45.202845 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.203691 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.205110 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.205652 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.205843 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:45.206504 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:45.207306 (+ 802us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:45.207427 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:45.207586 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 09:16:45.207662 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":484,"threads_started":1}
I20260806 09:16:45.229771 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:45.217138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.3.69.61:39859)
0806 09:16:45.217487 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:45.217494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:45.221642 (+ 4148us) server_negotiation.cc:407] Connection header received
0806 09:16:45.221857 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:45.221863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:45.221975 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:45.222102 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:45.222924 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.224264 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.226609 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.227351 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.227458 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:45.228870 (+ 1412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:45.229023 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:45.229093 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:45.229345 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 09:16:45.229448 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260806 09:16:45.237340 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:45.217375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.3.69.60:39921)
0806 09:16:45.217567 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:45.217572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:45.230570 (+ 12998us) server_negotiation.cc:407] Connection header received
0806 09:16:45.230899 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:45.230907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:45.231002 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:45.231066 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:45.231781 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.232883 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.235090 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.235746 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.235843 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:45.236497 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:45.236656 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:45.236721 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:45.236957 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 09:16:45.237050 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:16:45.240938 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:16:45.217673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.3.69.62:41969)
0806 09:16:45.229350 (+ 11677us) server_negotiation.cc:206] Beginning negotiation
0806 09:16:45.229360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:16:45.229389 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:16:45.229534 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:16:45.229542 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:16:45.229825 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:16:45.231306 (+ 1481us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:16:45.231554 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.233034 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.236210 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:16:45.237108 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:16:45.238230 (+ 1122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:16:45.239332 (+ 1102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:16:45.239923 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:16:45.240059 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 09:16:45.240532 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 09:16:45.240644 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11489,"thread_start_us":12455,"threads_started":1}
I20260806 09:16:45.246755 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4467
I20260806 09:16:45.248224 4560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:45.362887 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4467
W20260806 09:16:45.402909 4710 connection.cc:535] server connection from 127.0.0.1:58954 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 09:16:45.403352 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4575
I20260806 09:16:45.404706 4666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:45.520521 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4575
I20260806 09:16:45.548039 3348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6yYR2E/build/asan/bin/kudu pid 4686
I20260806 09:16:45.549242 4775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:16:45.658288 3348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6yYR2E/build/asan/bin/kudu with pid 4686
2026-08-06T09:16:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5297 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10747 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (386925 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST